The Resource for February 2022
Hello Reader, here is this month’s iRODS news and developments!
If you’re facing an issue with iRODS you’re not sure how to solve, please do drop me a line; if I’ve come across a solution or seen something relevant elsewhere, I’ll do my best to let you know. Or just drop me a mail to say ‘Hi’. Always nice to hear from people, particularly in these pandemic times!
I’d love your thoughts and feedback on how this newsletter could be better for you.
News
Save The Date for iRODS UGM 2022!
July 5-8 Leuven, Belgium. Look forward to seeing you there!
"We are excited to announce that the iRODS User Group Meeting 2022 will take place July 5-8 in Leuven, Belgium! We invite you to join us and our host KU Leuven to learn how iRODS is shaping the future of data management. Registration is coming soon!
The iRODS UGM 2022 will feature multiple sessions of talks, demos, interactive discussions, and networking opportunities. To kick off the event, the iRODS Consortium Development Team will provide Beginner and Advanced Training sessions. On the last day of the UGM, we will hold an informal troubleshooting session to allow users the chance to participate in conversations with the entire core development staff about their current or planned deployments."
TriRODS
Mini IO Gateway for iRODS - YouTube
"This talk introduces a new iRODS backend for MinIO, a scalable, secure, S3-compatible object storage for cloud users. The iRODS backend to MinIO allows users to configure their iRODS login to access their restricted data or anonymous access to public/shared data with an iRODS Ticket. By integrating with MinIO, iRODS users can leverage the user-friendly mode of data access provided by the MinIO user interface and the MinIO object store REST API."
iRODS Globus Connector 4.2.11.1 Released
"Faster for lots of small files."
eResearcrNZ Community Talk
"Want to learn more about what the #eResearchNZ community does? Join on Feb. 10 at 6 p.m. ET / Feb. 11 at 12 p.m. NZDT to hear David Fellinger’s #iRODS discussion “Data Management Technology Driven and Sustained by the eResearch Community”.
Over now of course, but you might be able to watch the recorded talk, I think?
Python iRODS Client v1.1.0 and 1.1.1 are released
It has a changelog!
iRODS Research Community Requirements Drive Expanded Scale Data Management Features Talk
"Several years ago, the entire process of data management and collaboration could only be performed with the use of proprietary software products that were expensive to license. To maintain a collection, data sites required a file system, hierarchical storage management system, and some means of sharing the data over several geographically diverse sites using purchased software, often from a single vendor to ensure compatibility. Data site managers were placed in a difficult position, facing quickly growing data capacity and transmission demands with limited budgets. Constraints from funding agencies and governments became very difficult, if not impossible, to manage and audit.
The iRODS (Integrated Rule-Oriented Data System) Consortium was started as an open-source software development organization in 2013 by members of the research and storage communities. The technology has roots from an earlier project started in 1995. The Consortium was launched as a response to a major scale increase in management and storage needs driven by the advent of "big data". The member community is now comprised of over 30 members and spans the globe from the Australia to Japan and much of the EU. Recent innovations as a result of community requirements will be discussed, including graphical interfaces and methods to ensure data persistence and replication management. In addition, partnerships will be discussed with Globus and others to enable large scale collaboration. Today, worldwide, FAIR discovery and directed dissemination of HPC results are being accomplished in sites controlling tens of petabytes of data with this open-source technology."
iRODS S3 Resource Plugin 4.2.11.1 is released
NFSRODS v2.1.0 is released
On a Friday night! Brave…
2.1.0
A fair number of changes worth pulling in, especially parallel transfer (if the client is multi-threaded) and logging.
Main Repository Activity
Open Issues
add delay server memory usage default to server_config.json on upgrade
FYI, If you are upgrading to 4.2.11;
"maximum_size_of_delay_queue_in_bytes should be added to the advanced_settings of server_config.json on upgrade if not already present. It should have a default value of 0 (unlimited)."
Weird packaging message for CentOS 7
4-2-stable, so 4.2.12 candidate?
rx_atomic_apply_acl_operations() Does not consider group permissions
rx_atomic_apply_metadata_operations() does not consider group permissions
Ticket tests do not cover rodsuser cases
Non-admins should not be allowed to run iadmin lg
I wonder how many iadmin commands can be run by rodsuser accounts?
Deprecate imeta adda and all APIs that support it
"After imeta grows support for the admin mode option (-M), the subcommand adda will not be needed because imeta -M add accomplishes the same thing."
Anyone’s workflow depend on imeta adda (which, naturally, reminds me of this?
iadmin rum Functionality can be invoked by non-admins
"Removing unused AVUs is an admin-level operation, as it affects an entire zone. Rodsusers cannot invoke this operation via the iadmin command because it is protected by the API lookup routines. However, a rodsuser can invoke the operation via irule and msiDeleteUnusedAVUs()."
I need to run this on my site - anyone used it on very large metadata catalogs?
Add support for admin mode to filesystem library metadata functions
Remove BOOST_ASIO_HAS_STD_INVOKE_RESULT macro once Boost can detect std::invoke_result
test_fixed_buffer_resource unit test fails
Expose some missing DataObjInfo members to RE serialization
imeta cp does not support relative paths
Investigate multiple calls to _rollback() in db_add_avu_metadata_op()
Consider appending “_atomic” to atomic filesystem API functions
test_all_rules no longer runs the auto-generated tests
Make GenQuery syntax closer to SQL
"GenQuery should be exactly like SQL, but without needing to specify any tables.
The statement above has been satisfied … however, when it comes to syntax, some of the choices lead to confusion and misunderstanding.
Perhaps the choices were due to hardware limitations. Regardless, machines are more powerful and programming languages are more capable, so let’s improve GenQuery.”
remote zone name can be an empty string
prevent renaming of remote zones with users defined
prevent deletion of remote zones with users defined
Review SQL queries used in db_plugin.cpp
Use “JOIN-ON” clause when joining tables in SQL
Add admin flag (-M) to imeta
Latest jsonschema version seems unhappy with iRODS JSON schemas
Seen with 4.0.0, 18.04 Ubuntu seems to have 2.6.0, so FYI if you need to upgrade it.
Build error on rocky Linux
fatal error when building in rocky Linux
Am I the only one thinking of Frank N Furter here? “I made you, so I can break you, too” (RIP Meat Loaf)…
Replicas remain in intermediate state when resource plugin close operation returns an error.
Appears to affect S3 plugin, not clear if it’s others.
Declare several common functions in header files
Investigate moving irods::experimental::authentication_base out of experimental namespace
Revisit Boost.Stacktrace backend implementation
Server will not start when run in a docker container emulating different architecture
I admit, I was thinking “well, who does THAT”, and then realised that Ubuntu on OSX is actually a pretty common Bioinformatician workflow…
Storage tier migration does not recover from the destination resource filling temporarily
As well as a lot of discussion around “Minimum Space For Create In Bytes” and how it’s checked vs freespace, this issue also has a nice set of queries to determine objects ‘in flight’ during migration. If I ever get it working as pure SQL, I’ll update the ticket…
Hidden requirement on multi-1247 parallel transfer against replication resource
"To resolve this issue, we have to claim that the requirement is part of the design, or we lift the requirement so that the last stream to close is responsible for updating the catalogue and triggering policy."
I’d say the latter, but perhaps I am missing something? Anyone else have any opinions and want to weigh into the issue?
Error messages in server log during successful storage tier migration
"And arguably, the storage tiering framework should drop some of those NOTICE messages to DEBUG... it's very talkative at the moment."
Organize and establish conventions for irods_configuration_keywords.hpp
iexit full can remove .irodsA file for service account
"Should an administrator get into this situation, iinit can be run as usual and the password entered to generate the.irodsA file again after waiting for the connection to fail. The server can then be restarted and things will return to normal."
Again, I am reminded of this
setting group ownership on resource write
A slightly clearer version of the title should have been “Configure iRODS to write file to its resources as a non-primary group.”
Re-implement irods_stacktrace.hpp/cpp in terms of Boost.Stacktrace
rodsadmin should not be allowed to downgrade user type of another rodsadmin operating an irods-server
rodsadmin can downgrade their own user type
Cannot remove expiration time from ticket (possible regression)
icommands don’t cope well with $PWD being deleted
Add support for ADMIN_KW to filesystem metadata API functions
DATA_OBJ_INFO.resc_id not being serialized correctly for pep_database_mod_data_obj_meta_post
Use obfEncodeByKeyV2 in user administration library
Closed Issues
Closed on - 2022-02-16 14:45:52 Incorrect name used to detect admin mode option in imeta
Closed on - 2022-02-18 14:46:43 Split up test_catalog
Closed on - 2022-02-15 16:02:15 Bump fmt dependency version in CMakeLists.txt
Closed on - 2022-02-15 16:02:22 Disable C++ compiler extensions
Closed on - 2022-02-17 17:53:14 Package unit tests
Closed on - 2022-02-06 00:59:27 irods_setup.py: “error while loading shared libraries: libboost_iostreams.so.1.77.0”
Closed on - 2022-02-02 23:09:10 Update logger to use clock_gettime() instead of gettimeofday()
Closed on - 2022-01-27 23:59:01 irods-server package is not declaring dependency on irods-externals-clang-runtime
Closed on - 2022-01-21 02:56:36 iCommands fail to build on main
Closed on - 2022-01-21 13:06:24 run_tests.py assumes python2-style filter()
Closed on - 2022-02-15 17:43:35 iquest result page length is no longer 500
Closed on - 2022-02-18 14:46:52 Prevent checksum verification in sync-to-archive operations
Closed on - 2022-02-15 16:40:12 cppzmq deprecations
Closed on - 2022-02-02 18:47:24 Do not create “rodsadmin” group on new installs
Closed on - 2022-01-27 21:32:56 nlohmann-json externals is packaged poorly
Closed on - 2021-12-21 18:38:36 Support for Ubuntu 20.04
… for 4.3.0 (but see issue for workarounds).
Closed on - 2022-02-15 17:41:54 iquest should return either zero or CAT_NO_ROWS_FOUND, not both
Closed on - 2022-01-25 13:00:20 Remove msiSetRescSortScheme
Deprecated back in 2013. Surely no one is using this now?
Closed on - 2022-01-10 19:47:39 containerize irods
Pertinent update - links to development and testing environments using Docker.
Closed on - 2022-02-10 16:39:16 Intermittent failure to connect in tests run on resource server
Python iRODS Client Activity
Open Issues
Update connection disconnect logic.
Enable cwd and chdir methods allowing ‘.’ and ‘..’
Allow parallel transfer to/from S3 storage
Fixed bug in clean-up of stale connections
Closed Issues
Closed on - 2022-02-19 03:42:30 failing tests need revision for minor relative-path concerns
Closed on - 2022-01-30 19:14:41 RE_RULE_ENGINE_ERROR doesn’t exist in exception module
Closed on - 2022-01-30 19:15:09 Add clarification for python rule execution in Readme
Closed on - 2022-01-30 19:14:29 Execute rule text as if it were contained in a rule file
Closed on - 2022-02-19 03:42:05 Use ‘-R’ to identify resource type and ‘-C’ to identify collection in AVU operations
Closed on - 2022-02-18 15:39:47 Trailing slash in path gives python irods error while the same path works using icommands
NFSRODS Activity
Open Issues
Write operations generate lots of log messages in iRODS server
make NFSRODS report via spOption
"This will make it appear correctly to ips for the iRODS administrator."
Closed Issues
I think a lot of these are addressed in 2.1.0 released above, but as the issues aren’t tagged to releases as the main iRODS repo is, I haven’t got back to the change log to verify.
Closed on - 2022-02-05 00:15:28 Bump Jargon version to 4.3.2.5-SNAPSHOT
Closed on - 2022-01-26 04:16:34 Remove installation and handling of rpcbind from docker-related files
Closed on - 2022-01-26 04:27:57 iRODS timestamps do not provide enough precision to detect changes
"It is possible for NFSRODS to present incorrect information when operations are executed very quickly.
NFS detects changes in the filesystem via the mtime. iRODS stores timestamps in seconds. Systems depending on these timestamps will not operate correctly if operations complete within the same second."
Closed on - 2022-01-26 04:15:28 Combine subsequent calls to logger
Closed on - 2022-01-26 04:14:38 password appearing in log
Closed on - 2022-01-26 04:15:03 Does not run without /dev/stdin bound to docker
"Commit 318663d makes running the docker image without the -i flag crash immediately, as /dev/stdin is absent. This makes a container workflow completely useless."
Closed on - 2022-01-26 04:14:26 Issue copying a file greater than 55Meg to nfsrods
Closed on - 2022-01-26 04:14:54 No log output on Ubuntu 18
Closed on - 2022-01-27 01:32:15 NFSRODS should support user and group auto-creation
Closed as wont fix - the system admin needs to manage the user/group state of the iRODS Zone outside of NFSRODS.
icommands Activity
Open Issues
build error in rocky with main branch
"The main branch of the irods/irods repo is out of sync with this repo."
…
"Once irods/irods#6156 is merged, you can re-enable/uncomment the lines mentioned above."
usr directory error
"I've just tried to build 4-2-stable non-package-install.... and it almost worked."
I haven't tried building the icommands via run_in_place for 4.2.11, so say on this issue if you have, and it doesn't work. If not, it's likely affecting what 4-2-stable branch, so should be fixed by 4.2.12 is released.
irepl does not process remaining existing data objects if it processes a non-existent one
Also involves a really nice discussion on how one might want to handle irepl when dealing with large data sets.
Install icommands on CentOS 8 (arch: x84_64) and Red Hat 8 (arch: ppc64le)
Closed Issues
externals Activity
Open Issues
Build error in Ubuntu 18.04
"I recommend looking at 4.2.11 documentation - it's mostly the same as 4.2.3 (linked), but some things have moved.
make sure you're building the 4-2-stable branch (git checkout 4-2-stable)
you can try make server first..
this subset of targets are the requirements for the irods-server.
additional plugins may require some of the other make targets."
Closed Issues
Closed on - 2022-02-04 19:06:41 Add "enable_sha": true to packages that use an SHA as the checkout target
Closed on - 2022-02-04 19:06:09 Fix compilation issues: munge fs does not compile with clang 13
Closed on - 2022-02-04 19:06:21 Remove qpid-cpp and qpid-with-proton packages
Closed on - 2022-02-07 12:32:52 build error in Rocky Linux
Closed on - 2022-02-04 21:16:07 Git clone only the version defined by versions.json
Closed on - 2022-02-04 21:14:31 Elasticlient build error: Duplicate Case Value
YODA Activity
Open Issues
[BUG] error message when you click on the “Update” button of a group without changing anything
[BUG] the second time a new user is added to a group, the link “Click here to add a new user to this group” disappears
[BUG] adding a category with the character “_” in the name prevents the creation of the group “datamanager-“
"We do not allow underscores in group names, only lowercase letters, numbers and hyphens. The datamanager group always has the same name as the category. When the category name contains an underscore the option to create a datamanager group is removed, because the datamanager group would be invalid."
[BUG] Unable to transfer files with ‘select’ anywhere in the filename
Upstream iRODS bug: irods/irods#4697
Closed Issues
Closed on - 2022-02-08 07:44:14 [FEATURE] consistent use of the word “user” or “member” in the portal
Closed on - 2022-02-08 07:33:55 [BUG] Can’t fill in (Sub)Category in Group Manager
Closed on - 2022-01-27 09:25:58 [FEATURE] davrods on different host than yoda
If you think someone else would appreciate this newsletter, they can sign up at https://theresource.metadata.school/
Four Yaks were shaved in the making of this newsletter.