Hello Reader, here is this month’s iRODS news and developments!

If you’re facing an issue with iRODS you’re not sure how to solve, please do drop me a line; if I’ve come across a solution or seen something relevant elsewhere, I’ll do my best to let you know. Or just drop me a mail to say ‘Hi’. Always nice to hear from people, particularly in these pandemic times!

I’d love your thoughts and feedback on how this newsletter could be better for you.

News

iRODS C++ REST API 0.9.0 is released

​iRODS C++ REST API​changes​

Seems to be mostly bumping (requiring?) for 4.2.11 and README updates.

iRODS Globus Connector 4.2.11.0 is released

​iRODS Globus Connector - changes​

Seems to be;

• multithreaded transfers
• use only good replicas
• supporting (requiring?) 4.2.11

iRODS Python Rule Engine Plugin 4.2.11.1 is released

​iRODS Python Rule Engine Plugin 4.2.11.1 is released - changes​

Seems to be addressing this bug​

iRODS Client NFSRODS 2.0.4 is released

​iRODS Client NFSRODS 2.0.4 - changes -this fixed the Log4J vulnerability, so a must-do upgrade.

iRODS Development Update: December 2021

This came out too late for the last newsletter - iRODS Development Update: December 2021​

iRODS 4.2.11 is released

​iRODS 4.2.11 is released​

This release consists of 120 commits from 9 contributors and closed 124 issues.

Upcoming

Word on the street is Kanki is being released.

4.2.12 Might not be too far away either?

Main Repository Activity

Open Issues

A lot of issues being raised by the team ahead of 4.2.12. I’ve left in things like library updates and CMAKE fixes, but not gone into any detail. Let me know if that’s the kind of thing you want to see, though…

I’ve also left out issues which aren’t new but had come up in my search because they had been tagged, or moved to a new version tag. Again, if that’s the kind of thing you would find value in seeing here, let me know.

​Replication from compound using python​

​rebalance should continue on data object that has since been moved out-of-tree​

In other words, if there are two more replicas and the bad one is in the tree, continue on with the rest of the rebalance, rather than stopping.

​Allow replication resource to create replicas on overwrite​

When a resource is marked down (or set to no write in weighting?), and a replica exists on that resource, it doesn’t get updated (which makes sense). New files will of course use different resources, but not update/overwrite operations, so this enhancement may allow updates to create new, good, replicas on other resources and then the bad replica can be updated by a rebalance at a later date.

​Microservices leak from outgoing msParam_t​

Memory leak; fixed in 4.2.12.

​Misleading error on wrong zone name​

​CMake: tweak target names, reorganize cmake configuration​

​CMake: Use CMAKE_CURRENT_SOURCE_DIR and CMAKE_CURRENT_BINARY_DIR where appropriate​

​CMake: Do not add libm to CMAKE_CXX_STANDARD_LIBRARIES​

​CMake: check compiler features​

​iquest not like operator not working ​

This only seems to be the case for columns that are generated by the query aka GenQuery.

​Populate (DEST_)RESC_HIER_STR_KW for DataObjInp.condInput in rsDataObjRepl/rsDataObjPhymv​

“In 4.2.9, rsDataObjRepl and rsDataObjPhymv were rewritten for clarity and correctness, which includes treating the DataObjInp as a pure input rather than an input/output.” - targeted for 4.2.12.

​CMake: use targets instead of variables for curl​

​iquest result page length is no longer 500​

“Probably been wrong since we fixed all the boundary issues in 4.1.11 4.5y ago”. Now around 255/256…

​Add register and unregister operations for weighted passthru​

​iunreg should be able to unregister a replica when the storage resource is down​

​Tip of “main” fails to compile unit tests​

Closed while I was assembling this post!

​Creation of 64 byte user names is allowed, even though they cause issues​

“If I can create a user with a 64 byte name, then such names should be supported by iRODS/iCommands.”

​Prevent checksum verification in sync-to-archive operations​

“When replication occurs in a sync-to-archive operation, a checksum verification will be performed for replicas whose source replica contains a checksum. This will fail for most actual archive resources with DIRECT_ARCHIVE_ACCESS, and the checksum from the source replica will be applied to the destination replica (even if it is not true).

The NO_COMPUTE_KW should be used when finalizing the replica to indicate that a checksum should not be computed for the destination replica. Should a checksum be desired, policy can be defined to calculate or simply apply a checksum value to the replica’s system metadata.”

​CMake: Use catch_discover_tests instead of add_test​

​catch2 externals package has include files in the wrong place.​

​cppzmq deprecations​

​Bump externals in jenkins CI hook scripts​

​Investigate pyodbc encoding/decoding function calls in database_connect.py​

For Ubuntu 20, If I am reading the bug report right.

​itree fails with exception on relative path ‘..’ or ‘../some/path’​

​Resource redirect is wrong if resource hostname contains current hostname as substring ​

Seems to mostly affect ibun but I may be misreading…

​Exception failure when talking cross federation to a client zone whose SSL certificate has expired​

​Server should sanitize all path inputs on entry​

​Fix remaining leaks in REPF/PREP​

​Fix delay hints parser for “DOUBLE” directives​

​nlohmann-json externals is packaged poorly​

This is under ongoing development - check out the number of commit’s related to it!

Closed Issues

Closed on - 2021-12-21 21:45:55 Specify which externals packages need to be installed for GitHub actions​

4.3.0…

Closed on - 2021-12-21 18:37:44 Bump Clang compiler version to 13​

4.3.0…

Closed on - 2021-12-21 18:36:16 Add support for Python 3​

4.3.0…

Closed on - 2021-12-21 18:37:07 Python3​

4.3.0…

Closed on - 2021-12-29 00:29:00 optimization/atomicity when manipulating resource hierarchies​

“We chose not to implement setparentresc and instead… just allowed overwriting with addchildtoresc.”

Closed on - 2021-12-21 19:27:07 irods-icommands installation fails on Ubuntu20​

Closed because support is now available for 4.3. Which…hasn’t been released yet.

Closed on - 2021-12-21 19:26:19 Request for icommands to be built for Ubuntu 20.04​

Closed because support is now available for 4.3. Which…hasn’t been released yet.

Closed on - 2021-12-21 18:38:36 Support for Ubuntu 20.04​

Closed because support is now available for 4.3. Which…hasn’t been released yet.

Closed on - 2022-01-10 19:47:39 containerize irods​

“No action here - our build and testing environments are now containerized.”

Which is great… but the rest of us are left to figure this out?

Some documentation on how to do this/things to avoid, in the docs would help, I feel, even if a container image is too far to stretch at the moment.

Closed on - 2022-01-10 20:28:31 rodslog rotation didn’t​

4.3.0 uses syslog. All hail syslog!

Closed on - 2022-01-10 20:58:55 refactor irods shared libraries​

Addressed in 4.2.8, apparently.

Closed on - 2022-01-10 21:02:13 Bump CMake required version and externals version to 3.7.0 once it is released​

Closed on - 2022-01-10 21:02:47 Document new irods shared libraries for use by clients​

“Out of date. Closing.”

Closed on - 2022-01-10 21:04:27 building iRODS 4 without root and without using or requiring OS packages​

“Closing - this has worked for a few years now.”

Closed on - 2022-01-10 21:05:27 oracle odbc segfault _cllExecSqlNoResult​

“Our supported OS list does not include these anymore - closing.” (referring to Ubuntu 12)

Closed on - 2022-01-10 21:07:07 ilsresc -l cross-version failure​

“Closing - we’ve seen federation with 3.x work for a while.”

Closed on - 2022-01-15 19:57:28 define the rebalance operation for random and roundrobin resource plugins​

“The roundrobin resource has been marked deprecated in 4.2.x (#3778) and removed for 4.3.0 (#3779).

The rebalance operation for the random resource is a no-op.

I don’t think there is anything to do here. Closing.”

Closed on - 2022-01-10 21:22:20 icommands: ensure error messages are printed, and return codes are !=0, if operations are not entirely successful​

After a number of commits tagged to the issue;

“Closing - if we need more specific fixes, we can open new issues.”

Closed on - 2022-01-10 21:31:52 federation testing​

Federation tests added a while back.

Closed on - 2022-01-11 10:40:12 Add libcurl3-gnutls as package dependency on Debian​

Old Debian dependency on libCurl, removed a while back.

Closed on - 2022-01-10 21:43:50 Remove leftover Database Resource code​

Closed on - 2022-01-10 21:52:23 release HDFS plugin 1.0​

“Not going to release Hadoop File System plugin.” - I had forgotten there was one of these! No interest from the community, it seems?

Python iRODS Client Activity

Open Issues

​Need a secure option for parsing iRODS XML protocol​

“In #325 Codacy flagged the use of xml.etree.ElementTree as a security vulnerability due to the ability of various exploits to occupy large amounts of memory due by leveraging parser features”.

Although not tagged as such in the issue, it was part of the v1.1.0 release

​Use ‘-R’ to identify resource type and ‘-C’ to identify collection in AVU operations​

“Maybe I should go ahead and put through the change I’d planned before, and see if the old problem with -c/C, -r/R can be reproduced. Or at least make sure the python iRODS client does things consistently with the icommands.”

​Write proper tests around issue #279 (Pam Long Tokens)​

​Allow a rodsuser to change passwords à la “ipasswd”​

“There is not currently a way for users to change their own password in the iRODS database.”

Closed Issues

Closed on - 2022-01-15 22:28:07 Ticket troubles​

TL;DR - if you are seeing this, make sure you have v1.1.0 installed.

‘The ticket “functionality” in v1.0.0 is near-null in terms of added value. Tbh we should’ve released a new PRC by now or at least made it clear in the README that the discussed Ticket functionality applies only to the forthcoming release v1.1.0 (due within the week)’

Closed on - 2022-01-14 14:03:09 Long PAM password/token string causes PACKSTRUCT error​

Incorporated in 1.1.0.

NFSRODS Activity

Open Issues

​Remove installation and handling of rpcbind from docker-related files​

“rpcbind is not needed inside of the Ubuntu 18 docker image.”

​iRODS timestamps do not provide enough precision to detect changes​

“It is possible for NFSRODS to present incorrect information when operations are executed very quickly.

NFS detects changes in the file system via the mtime. iRODS stores timestamps in seconds. Systems depending on these timestamps will not operate correctly if operations complete within the same second.

The following operations are known to be affected:

• unlink
• list
• rename/move
• copy
• write

Solution

To get around this, NFSRODS must manually clear caches on completion of these operations.”

Externals Activity

Open Issues

​build error in Rocky Linux​

Closed Issues

Closed on - 2022-01-03 22:06:08 Boost cmake configuration files do not work with Python 3.10+​

This is on main branch.

“We know that make package is working… this is the subset of software in externals needed to build the server and the icommands.”

YODA Activity

Open Issues

​[BUG] Starting irods fails during 1.8 deployment​

“Looks like it’s related to switching to systemd used by irods-4.2.11”

“After running irodsctl stop once, the service comes back up fine and the issue does not occur on subsequent plays. Perhaps add an extra task to first check for irods-4.2.7 and irodsctl stop in the irods_icat role?”

If you think someone else would appreciate this newsletter, they can sign up at https://theresource.metadata.school/​

Two Yaks were shaved in the making of this newsletter.