I’m currently making use of TileDB as part of an AWS Batch array job writing out fragments to S3. TileDB was built without verbosity, but I’m seeing messages printed in the Cloud Watch logs related to every data transfer that happens to S3. How would I go about silencing these request/response messages since they are drowning out my own application’s debug/trace info in those logs?
There were some debugging statements in AWS SDK code. We submitted a fix upstream, but haven’t updated the SDK version we use since they merged the PR:
I’ll bump our version now and see what happens on CI…
Thanks, I’ll keep an eye out for updates.
@AndyGreenwell version is bumped on dev, which eliminates the debug output:
(currently the superbuild does not really support dependency updates, so it is probably safest/fastest to do a clean build)
Thanks @ihnorton.
I just pulled the latest dev branch, recreated the build folder, executed:
../bootstrap --enable-s3 --enable-static-tiledb --prefix=/usr/local
and built using make from within an Amazon Linux 2 VM.
The overall build failed at the point of building libcurl, with the following entry in the externals/src/ep/ep_curl-stamp/ep_curl-build-err.log file:
vtls/openssl.c: In function "Curl_ossl_seed":
vtls/openssl.c:428:15: error: implicit declaration of function "RAND_egd"; did you mean "RAND_add"? [-Werror=implicit-function-declaration]
int ret = RAND_egd(data->set.str[STRING_SSL_EGDSOCKET]?
^~~~~~~~
RAND_add
cc1: some warnings treated as errors
make[5]: *** [vtls/libcurl_la-openssl.lo] Error 1
make[4]: *** [all] Error 2
make[3]: *** [all-recursive] Error 1
I can see you are already dealing with build issues here, so I will keep an eye on that issue.
What version of openssl are you using? It appears that RAND_egd does not exist in openssl v1.1.0+. You could work around this by downgrading to an earlier version. I am able to compile with v1.0.2.
My instance of Amazon Linux 2 that I have installed in my local VirtualBox instance has openssl v1.0.2k installed in /usr/lib64 which seems to be what is being picked up by the bootstrap script.
I am attempting to statically compile all dependencies into the package where I am using TileDB, but apparently was not pointing the TileDB build process at my own statically compiled version of that library (1.1.1b). I was pointing the compilation of my python package at that static lib though.
I will try rebuilding and point the bootstrap script at a specific instance of OpenSSL that is different from the v1.0.2k instance that the bootstrap script is finding.
Curl supports both versions, but perhaps
- the detection logic was flawed in the version we currently use for
dev - it is looking at an older version if AWS linux has them side-by-side
- you need a different set of headers for a static build (not sure if the system path provides that by default)
We are going to bump the curl version shortly, which may help:
However, since you want a static build I would suggest to consider using bootstrap .. --force-build-all-deps which will build openssl from source and link it statically (I tested this against the AWS version bump with CentOS5 last night).
In attempting to use --force-build-all-deps with the full command used being:
../bootstrap --enable-s3 --enable-static-tiledb --prefix=/usr/local --force-build-all-deps
and then building with make, I am getting an error related to finding AWSSDK::pthread.
-- Found AWS lib: pthread (AWS_FOUND_pthread-NOTFOUND)
CMake Error at cmake/Modules/TileDBCommon.cmake:85 (message):
Could not determine library location for AWSSDK::pthread
Call Stack (most recent call first):
cmake/Modules/FindAWSSDK_EP.cmake:125 (install_target_libs)
tiledb/CMakeLists.txt:272 (find_package)
-- Configuring incomplete, errors occurred!
See also "/home/ec2-user/TileDB/TileDB/build/tiledb/CMakeFiles/CMakeOutput.log".
make[2]: *** [tiledb-prefix/src/tiledb-stamp/tiledb-configure] Error 1
make[1]: *** [CMakeFiles/tiledb.dir/all] Error 2
make: *** [all] Error 2
The libpthread.so library is located in /usr/lib64/libpthread.so on this Amazon Linux 2 VM.
I see there was a long issue discussion on the AWS SDK repo (issue 1020) about building cleanly with all dependencies, so I’ll try to follow that discussion to determine if there is something off about my environment in attempting to perform this build.
Thanks again for the assistance.
p.s. I tried posting a link to the aws-sdk-cpp GitHub repo in this reply, but the system prevented posting the reply with that link included.
Fixed by: https://github.com/TileDB-Inc/TileDB/pull/1240/commits/5b48cb422435321c665fea75a0cb573d2e37e375 (not merged yet, waiting for CI on other platforms)
Now I get a clean build with bootstrap --enable-static-tiledb --force-build-all-deps --enable-s3 (my direct cmake invocation wasn’t equivalent)
I suspect you may also hit https://github.com/TileDB-Inc/TileDB/issues/1080, which I/we won’t be able to fix right now – but it should be straightforward to work around by adding the libraries listed by curl config to your flags.
I guess I am being a bit dense here (probably due to lack of cmake expertise), but I have tried and failed at adding the libraries listed by curl config in two different ways that seemed as if it might be appropriate.
In one attempt, I listed each of the static libraries that I built myself as a colon separated list to the --dependency=DIRs flag to ./bootstrap. That attempt was a few hours ago, and (while I don’t remember the specific failure) that attempt was not successful.
In a second attempt, I edited cmake/Modules/FindCurl_EP.make (line 79) to add each of the -L and -l entries listed from curl-config as stated in the above referenced issue 1080. In that attempt, building with --enable-static-tiledb --force-build-all-deps compiled successfully, but then when built into my python module, I received the following error upon attempting to load the module:
ImportError: /usr/local/lib64/python3.7/site-packages/mymodule.cpython-37m-x86_64-linux-gnu.so: undefined symbol: HMAC_CTX_reset
That symbol is from OpenSSL, which leads me to believe I have not linked these libraries properly at some point in the build chain (either building TileDB or building mymodule). Since I had previously been able to build TileDB (without using the --force-build-all-deps) and load it into my module successfully, I believe the contents of my setup.py file are currently sufficient.
If you would be willing to be specific in exactly what needs to be added to the referenced flags to bootstrap or what file might need editing in order to statically compile libcurl.a with all of its dependencies as part of this build process, then I am happy to work from my own fork/branch for now just to get my application into an operational state on AWS.
Thanks again.
p.s. Attempting to include another couple of links to GitHub flagged my post as spam again.
Hi Andy,
First, this build error:
is fixed by this PR I just merged:
The curl version bump should also fix this issue with OpenSSL detection:
So, I am hopeful that your original build script should work with the latest dev.
That said, if the above doesn’t work easily, then my other suggestion to “get things moving” would be to go back to your original, working build script and TileDB version, and apply this patch locally to your AWS SDK build: Remove extraneous std::cout print statements in request path by joe-maley · Pull Request #1108 · aws/aws-sdk-cpp · GitHub – that will remove the AWS debugging printouts. You can find the source in /PATH/TO/tiledb-build/externals/src/ep_awssdk. Then do
cd /PATH/TO/tiledb-build/externals/src/ep_awssdk-build
make install
Then re-link your extension.
Hope that helps,
Isaiah
ps: sorry about the link restrictions/flags… @jakebolewski said he made the Discourse settings more lenient, but we may still need to tweak the settings a little bit.
Back in business, and using your most recent commit to dev.
I’m currently statically compiling a number of the dependencies for libcurl.a myself (libnghttp2.a, libidn2.a, libunistring.a) and within the build of my python module linking those with the static output of the various libraries from the Superbuild (ensuring proper ordering of all libraries in the extra_link_args entries).
Thanks again,
Andy
1 Like
That’s great, and it’s good to know that statically linking the extension works, because we might want to do that in the future to simplify packaging.