The Resource logo

The Resource

Archives
Subscribe
February 21, 2022

The Resource for February 2022

Hello Reader, here is this month’s iRODS news and developments!

If you’re facing an issue with iRODS you’re not sure how to solve, please do drop me a line; if I’ve come across a solution or seen something relevant elsewhere, I’ll do my best to let you know. Or just drop me a mail to say ‘Hi’. Always nice to hear from people, particularly in these pandemic times!

I’d love your thoughts and feedback on how this newsletter could be better for you.

News

Save The Date for iRODS UGM 2022!

​July 5-8 Leuven, Belgium. Look forward to seeing you there!

"We are excited to announce that the iRODS User Group Meeting 2022 will take place July 5-8 in Leuven, Belgium! We invite you to join us and our host KU Leuven to learn how iRODS is shaping the future of data management. Registration is coming soon!
The iRODS UGM 2022 will feature multiple sessions of talks, demos, interactive discussions, and networking opportunities. To kick off the event, the iRODS Consortium Development Team will provide Beginner and Advanced Training sessions. On the last day of the UGM, we will hold an informal troubleshooting session to allow users the chance to participate in conversations with the entire core development staff about their current or planned deployments."

TriRODS

​Mini IO Gateway for iRODS - YouTube​

"This talk introduces a new iRODS backend for MinIO, a scalable, secure, S3-compatible object storage for cloud users. The iRODS backend to MinIO allows users to configure their iRODS login to access their restricted data or anonymous access to public/shared data with an iRODS Ticket. By integrating with MinIO, iRODS users can leverage the user-friendly mode of data access provided by the MinIO user interface and the MinIO object store REST API."

iRODS Globus Connector 4.2.11.1 Released

"Faster for lots of small files."

​4.2.11 - changes​

eResearcrNZ Community Talk

"Want to learn more about what the #eResearchNZ community does? Join on Feb. 10 at 6 p.m. ET / Feb. 11 at 12 p.m. NZDT to hear David Fellinger’s #iRODS discussion “Data Management Technology Driven and Sustained by the eResearch Community”.

Over now of course, but you might be able to watch the recorded talk, I think?

Python iRODS Client v1.1.0 and 1.1.1 are released

​1.1.1 - changes​

It has a changelog!

​1.1.0 - changes​

iRODS Research Community Requirements Drive Expanded Scale Data Management Features Talk

​Terrell’s Talk​

"Several years ago, the entire process of data management and collaboration could only be performed with the use of proprietary software products that were expensive to license. To maintain a collection, data sites required a file system, hierarchical storage management system, and some means of sharing the data over several geographically diverse sites using purchased software, often from a single vendor to ensure compatibility. Data site managers were placed in a difficult position, facing quickly growing data capacity and transmission demands with limited budgets. Constraints from funding agencies and governments became very difficult, if not impossible, to manage and audit.
The iRODS (Integrated Rule-Oriented Data System) Consortium was started as an open-source software development organization in 2013 by members of the research and storage communities. The technology has roots from an earlier project started in 1995. The Consortium was launched as a response to a major scale increase in management and storage needs driven by the advent of "big data". The member community is now comprised of over 30 members and spans the globe from the Australia to Japan and much of the EU. Recent innovations as a result of community requirements will be discussed, including graphical interfaces and methods to ensure data persistence and replication management. In addition, partnerships will be discussed with Globus and others to enable large scale collaboration. Today, worldwide, FAIR discovery and directed dissemination of HPC results are being accomplished in sites controlling tens of petabytes of data with this open-source technology."

iRODS S3 Resource Plugin 4.2.11.1 is released

​4.2.11.1 - changes​

NFSRODS v2.1.0 is released

On a Friday night! Brave…

​2.1.0​

A fair number of changes worth pulling in, especially parallel transfer (if the client is multi-threaded) and logging.

Main Repository Activity

Open Issues

​add delay server memory usage default to server_config.json on upgrade​

FYI, If you are upgrading to 4.2.11;

"maximum_size_of_delay_queue_in_bytes should be added to the advanced_settings of server_config.json on upgrade if not already present. It should have a default value of 0 (unlimited)."

​Weird packaging message for CentOS 7​

4-2-stable, so 4.2.12 candidate?

​rx_atomic_apply_acl_operations() Does not consider group permissions​

​rx_atomic_apply_metadata_operations() does not consider group permissions​

​Ticket tests do not cover rodsuser cases​

​Non-admins should not be allowed to run iadmin lg​

I wonder how many iadmin commands can be run by rodsuser accounts?

​Deprecate imeta adda and all APIs that support it​

"After imeta grows support for the admin mode option (-M), the subcommand adda will not be needed because imeta -M add accomplishes the same thing."

Anyone’s workflow depend on imeta adda (which, naturally, reminds me of this?

​iadmin rum Functionality can be invoked by non-admins​

"Removing unused AVUs is an admin-level operation, as it affects an entire zone. Rodsusers cannot invoke this operation via the iadmin command because it is protected by the API lookup routines. However, a rodsuser can invoke the operation via irule and msiDeleteUnusedAVUs()."

I need to run this on my site - anyone used it on very large metadata catalogs?

​Add support for admin mode to filesystem library metadata functions​

​Remove BOOST_ASIO_HAS_STD_INVOKE_RESULT macro once Boost can detect std::invoke_result​

​test_fixed_buffer_resource unit test fails​

​Expose some missing DataObjInfo members to RE serialization​

​imeta cp does not support relative paths​

​Investigate multiple calls to _rollback() in db_add_avu_metadata_op()​

​Consider appending “_atomic” to atomic filesystem API functions​

​test_all_rules no longer runs the auto-generated tests​

​Make GenQuery syntax closer to SQL​

"GenQuery should be exactly like SQL, but without needing to specify any tables.

The statement above has been satisfied … however, when it comes to syntax, some of the choices lead to confusion and misunderstanding.

Perhaps the choices were due to hardware limitations. Regardless, machines are more powerful and programming languages are more capable, so let’s improve GenQuery.”

​remote zone name can be an empty string​

​prevent renaming of remote zones with users defined​

​prevent deletion of remote zones with users defined​

​Review SQL queries used in db_plugin.cpp​

​Use “JOIN-ON” clause when joining tables in SQL​

​Add admin flag (-M) to imeta​

​Latest jsonschema version seems unhappy with iRODS JSON schemas​

Seen with 4.0.0, 18.04 Ubuntu seems to have 2.6.0, so FYI if you need to upgrade it.

​Build error on rocky Linux​

​fatal error when building in rocky Linux​

Am I the only one thinking of Frank N Furter here? “I made you, so I can break you, too” (RIP Meat Loaf)…

​Replicas remain in intermediate state when resource plugin close operation returns an error.​

Appears to affect S3 plugin, not clear if it’s others.

​Declare several common functions in header files​

​Investigate moving irods::experimental::authentication_base out of experimental namespace​

​Revisit Boost.Stacktrace backend implementation​

​Server will not start when run in a docker container emulating different architecture​

I admit, I was thinking “well, who does THAT”, and then realised that Ubuntu on OSX is actually a pretty common Bioinformatician workflow…

​Storage tier migration does not recover from the destination resource filling temporarily​

As well as a lot of discussion around “Minimum Space For Create In Bytes” and how it’s checked vs freespace, this issue also has a nice set of queries to determine objects ‘in flight’ during migration. If I ever get it working as pure SQL, I’ll update the ticket…

​Hidden requirement on multi-1247 parallel transfer against replication resource​

"To resolve this issue, we have to claim that the requirement is part of the design, or we lift the requirement so that the last stream to close is responsible for updating the catalogue and triggering policy."

I’d say the latter, but perhaps I am missing something? Anyone else have any opinions and want to weigh into the issue?

​Error messages in server log during successful storage tier migration​

"And arguably, the storage tiering framework should drop some of those NOTICE messages to DEBUG... it's very talkative at the moment."

​Organize and establish conventions for irods_configuration_keywords.hpp​

​iexit full can remove .irodsA file for service account​

"Should an administrator get into this situation, iinit can be run as usual and the password entered to generate the.irodsA file again after waiting for the connection to fail. The server can then be restarted and things will return to normal."

Again, I am reminded of this​

​setting group ownership on resource write​

A slightly clearer version of the title should have been “Configure iRODS to write file to its resources as a non-primary group.”

​Re-implement irods_stacktrace.hpp/cpp in terms of Boost.Stacktrace​

​rodsadmin should not be allowed to downgrade user type of another rodsadmin operating an irods-server​

​rodsadmin can downgrade their own user type​

​Cannot remove expiration time from ticket (possible regression)​

​icommands don’t cope well with $PWD being deleted​

​Add support for ADMIN_KW to filesystem metadata API functions​

​DATA_OBJ_INFO.resc_id not being serialized correctly for pep_database_mod_data_obj_meta_post​

​Use obfEncodeByKeyV2 in user administration library​

Closed Issues

Closed on - 2022-02-16 14:45:52 Incorrect name used to detect admin mode option in imeta​

Closed on - 2022-02-18 14:46:43 Split up test_catalog​

Closed on - 2022-02-15 16:02:15 Bump fmt dependency version in CMakeLists.txt​

Closed on - 2022-02-15 16:02:22 Disable C++ compiler extensions​

Closed on - 2022-02-17 17:53:14 Package unit tests​

Closed on - 2022-02-06 00:59:27 irods_setup.py: “error while loading shared libraries: libboost_iostreams.so.1.77.0”​

Closed on - 2022-02-02 23:09:10 Update logger to use clock_gettime() instead of gettimeofday()​

Closed on - 2022-01-27 23:59:01 irods-server package is not declaring dependency on irods-externals-clang-runtime​

Closed on - 2022-01-21 02:56:36 iCommands fail to build on main​

Closed on - 2022-01-21 13:06:24 run_tests.py assumes python2-style filter()​

Closed on - 2022-02-15 17:43:35 iquest result page length is no longer 500​

Closed on - 2022-02-18 14:46:52 Prevent checksum verification in sync-to-archive operations​

Closed on - 2022-02-15 16:40:12 cppzmq deprecations​

Closed on - 2022-02-02 18:47:24 Do not create “rodsadmin” group on new installs​

Closed on - 2022-01-27 21:32:56 nlohmann-json externals is packaged poorly​

Closed on - 2021-12-21 18:38:36 Support for Ubuntu 20.04​

… for 4.3.0 (but see issue for workarounds).

Closed on - 2022-02-15 17:41:54 iquest should return either zero or CAT_NO_ROWS_FOUND, not both​

Closed on - 2022-01-25 13:00:20 Remove msiSetRescSortScheme​

Deprecated back in 2013. Surely no one is using this now?

Closed on - 2022-01-10 19:47:39 containerize irods​

​Pertinent update - links to development and testing environments using Docker.

Closed on - 2022-02-10 16:39:16 Intermittent failure to connect in tests run on resource server​

Python iRODS Client Activity

Open Issues

​Update connection disconnect logic. ​

​Enable cwd and chdir methods allowing ‘.’ and ‘..’ ​

​Allow parallel transfer to/from S3 storage​

​Fixed bug in clean-up of stale connections​

Closed Issues

Closed on - 2022-02-19 03:42:30 failing tests need revision for minor relative-path concerns​

Closed on - 2022-01-30 19:14:41 RE_RULE_ENGINE_ERROR doesn’t exist in exception module​

Closed on - 2022-01-30 19:15:09 Add clarification for python rule execution in Readme​

Closed on - 2022-01-30 19:14:29 Execute rule text as if it were contained in a rule file​

Closed on - 2022-02-19 03:42:05 Use ‘-R’ to identify resource type and ‘-C’ to identify collection in AVU operations​

Closed on - 2022-02-18 15:39:47 Trailing slash in path gives python irods error while the same path works using icommands​

NFSRODS Activity

Open Issues

​Write operations generate lots of log messages in iRODS server​

​make NFSRODS report via spOption​

"This will make it appear correctly to ips for the iRODS administrator."

Closed Issues

I think a lot of these are addressed in 2.1.0 released above, but as the issues aren’t tagged to releases as the main iRODS repo is, I haven’t got back to the change log to verify.

Closed on - 2022-02-05 00:15:28 Bump Jargon version to 4.3.2.5-SNAPSHOT​

Closed on - 2022-01-26 04:16:34 Remove installation and handling of rpcbind from docker-related files​

Closed on - 2022-01-26 04:27:57 iRODS timestamps do not provide enough precision to detect changes​

"It is possible for NFSRODS to present incorrect information when operations are executed very quickly.
NFS detects changes in the filesystem via the mtime. iRODS stores timestamps in seconds. Systems depending on these timestamps will not operate correctly if operations complete within the same second."

Closed on - 2022-01-26 04:15:28 Combine subsequent calls to logger​

Closed on - 2022-01-26 04:14:38 password appearing in log​

Closed on - 2022-01-26 04:15:03 Does not run without /dev/stdin bound to docker​

"Commit 318663d makes running the docker image without the -i flag crash immediately, as /dev/stdin is absent. This makes a container workflow completely useless."

Closed on - 2022-01-26 04:14:26 Issue copying a file greater than 55Meg to nfsrods​

Closed on - 2022-01-26 04:14:54 No log output on Ubuntu 18​

Closed on - 2022-01-27 01:32:15 NFSRODS should support user and group auto-creation​

Closed as wont fix - the system admin needs to manage the user/group state of the iRODS Zone outside of NFSRODS.

icommands Activity

Open Issues

​build error in rocky with main branch​

"The main branch of the irods/irods repo is out of sync with this repo."

…

"Once irods/irods#6156 is merged, you can re-enable/uncomment the lines mentioned above."

​usr directory error​

"I've just tried to build 4-2-stable non-package-install.... and it almost worked."
I haven't tried building the icommands via run_in_place for 4.2.11, so say on this issue if you have, and it doesn't work. If not, it's likely affecting what 4-2-stable branch, so should be fixed by 4.2.12 is released.

​irepl does not process remaining existing data objects if it processes a non-existent one​

Also involves a really nice discussion on how one might want to handle irepl when dealing with large data sets.

​Install icommands on CentOS 8 (arch: x84_64) and Red Hat 8 (arch: ppc64le)​

Closed Issues

externals Activity

Open Issues

​Build error in Ubuntu 18.04​

"I recommend looking at 4.2.11 documentation - it's mostly the same as 4.2.3 (linked), but some things have moved.
make sure you're building the 4-2-stable branch (git checkout 4-2-stable)
you can try make server first.. 
this subset of targets are the requirements for the irods-server. 
additional plugins may require some of the other make targets."

Closed Issues

Closed on - 2022-02-04 19:06:41 Add "enable_sha": true to packages that use an SHA as the checkout target​

Closed on - 2022-02-04 19:06:09 Fix compilation issues: munge fs does not compile with clang 13​

Closed on - 2022-02-04 19:06:21 Remove qpid-cpp and qpid-with-proton packages​

Closed on - 2022-02-07 12:32:52 build error in Rocky Linux​

Closed on - 2022-02-04 21:16:07 Git clone only the version defined by versions.json​

Closed on - 2022-02-04 21:14:31 Elasticlient build error: Duplicate Case Value​

YODA Activity

Open Issues

​[BUG] error message when you click on the “Update” button of a group without changing anything​

​[BUG] the second time a new user is added to a group, the link “Click here to add a new user to this group” disappears​

​[BUG] adding a category with the character “_” in the name prevents the creation of the group “datamanager-“​

"We do not allow underscores in group names, only lowercase letters, numbers and hyphens. The datamanager group always has the same name as the category. When the category name contains an underscore the option to create a datamanager group is removed, because the datamanager group would be invalid."

​[BUG] Unable to transfer files with ‘select’ anywhere in the filename​

Upstream iRODS bug: irods/irods#4697

Closed Issues

Closed on - 2022-02-08 07:44:14 [FEATURE] consistent use of the word “user” or “member” in the portal​

Closed on - 2022-02-08 07:33:55 [BUG] Can’t fill in (Sub)Category in Group Manager​

Closed on - 2022-01-27 09:25:58 [FEATURE] davrods on different host than yoda​

If you think someone else would appreciate this newsletter, they can sign up at https://theresource.metadata.school/​

Four Yaks were shaved in the making of this newsletter.

Don't miss what's next. Subscribe to The Resource:
Powered by Buttondown, the easiest way to start and grow your newsletter.