You say you need to manage the memory yourself, so I'm going to believe you. I believe this completely eliminates garbage collected languages from consideration*, and that's most languages these days. Aside from C/C++, I think you still have Rust, D, and assembly as contenders. You might be able to use Java with with sun.misc.Unsafe, but that is not really recommended.
Not knowing too much about these languages, I would tend to think Rust might be a good choice. People seem to like it. I believe legacy code bases are frequently a big consideration in the decision to use C++.
And as others have said, you might be able to write libraries in C++ and include them in a code base that uses a higher level language, like Python.
*Are there languages that allow garbage collection and manual memory management? Not sure. I believe the answer is no.