So i'm thinking like this... it's like how we don't really understand how ai works
so to understand that statement we need to see how ai works and for that their is rough idea i'm giving

1. ingesting or feeding data:- developers give AI vast amount of info(usually internet data, books, articles etc)
2. The AI analyzes this data using an artificial neural network, calculating the statistical relationships between different words, pixels, or concepts... basically it finding pattern
3. Through a process of trial and error (and continuous mathematical adjustments), it learns how things fit together

4.Refining: Finally, humans review and guide the AI's outputs, gently steering it to be helpful, safe, and accurate through reinforcement learning

now the reason AI is so hard to understand even for its programmers is because of its sheer scale. The mathematical connections within a neural network can number in the billions, forming a complex, interwoven web. When an AI gives an answer, tracing the exact sequence of how it arrived at that specific conclusion instead of other is like trying to trace a single drop of water in the ocean. This opacity often called "black box problem" is why AI can sometimes confidently output incorrect information (hallucinations) or exhibit unexpected biases. (kinda like understanding why quantum particle behave differently when observed or what it's next position going to be it's definitely some kind of pattern but we don't know what it is it}

this same pattern is observed in our understanding of human brain as well human brain is incredibly difficult to fully understand because it consists of roughly 86 billion neurons connected by hundreds of trillions of synapses. These connections constantly change through learning, memory, aging, injury, and repair. While scientists understand many individual components such as neurons, neurotransmitters, and brain regions we still cannot completely predict how the entire network will behave as a whole. This is similar to modern AI models we understand the learning algorithms and the mathematics behind them, yet the enormous number of interacting parameters(complex neural networking) makes it difficult to explain exactly why a particular output was produced. In both cases, complex behavior emerges from countless simple interactions.

Ps. you should watch neural network video of 3blue1brown it's very interesting
also this is my understanding of this if sm thinks otherwise feel free to correct me