​

Hello Reader, here is this month’s iRODS news and developments!

If you’re facing an issue with iRODS you’re not sure how to solve, please do drop me a line; if I’ve come across a solution or seen something relevant elsewhere I’ll do my best to let you know. Or just drop me a mail to say ‘Hi’. Always nice to hear from people, particularly in these pandemic times!

I’d love your thoughts and feedback on how this newsletter could be better for you.

News

• ​iRODS Development Update February 2021​
• ​Virtual iRODS User Group Meeting 2021 - call for proposals - Wohoo! Will you be giving a talk? Why not try pitching one? I’m sure you’re doing something interesting…
• ​Why yes, this does mean pre-registration for the UGM 2021 is open! June 8-11th​
• ​CyVerse have a FUSE interface to iRODS - might not support SSL?
• ​MetalNX Roadmap as been released​
• ​Utrecht University have released iswitch - easy switching between iRODS iCommands configurations​

Main Repository Activity

open Issues

​Object remains in intermediate state when close operation returns an error.​

This is against 4-2-stable, so hopefully will be fixed by the time 4.2.9 is released.

​Method for non rodsadmin/groupadmin accounts to view group memberships across Federation​

TL;DR its currently only possible to list users in groups on another Zone using iquest rather than iuserinfo or igroupadmin as they lack the -z flag.

​On leaf-of-tree resource which is down, imv / irm succeed with error​

​iqstat -a does not return all rules when there are too many rules​

How many rules queued to run is too many? That’s an ecumenical question, but as for 4.2.8 and lower it appears around 11 million iqstat becomes a little unrelaiable…

​maximum_number_of_concurrent_rule_engine_server_processes seems not to work​

Related to the above, 4.2.9 will implement this limit.

​optimization/atomicity when manipulating resource hierarchies​

If you want to do compilcated tree surgery, say moving several filesystem resoucres into a different configuration under a different passthru, this will be very helpful.

​Refactor authentication plugin interface leveraging Authentication Working Group design​

Initial work from the Consrtium’s Working Group on Authentication.

​ibun reporting & documentation of errors handing symlinks​

​Consistently getting an error with replication​

Rather than a general replication issue, this is a report from a site which is having some issues with a fairly complex rulebase and setup. Might be quite niche, follow the issue if you want to find out more, I guess!

​irmtrash –orphan (no -M) silently does nothing​

​iuserinfo incorrect parsing of default username​

Basically, iuserinfo and iuserinfo use fifferent code paths, and while they should behave the same, dont. As different libraries are used for parsing, some characters, such as ‘@’ can cause issues.

​Overwriting and deleting files in mounted collections is broken in iRODS 4.2.8​

Well worth a read if you or your users use imcoll.

​Use irods::filesystem status information in lsUtil beyond collection/data object determination​

​Cleanup S3 shared memory on server startup and shutdown​

​Running icommands with new user throws errors but still works​

I suspect local resolution errors, but I could be wrong…

​noisy logs - caught exception when replicating to a resource which doesn’t exist​

​Caught exception caused by running two msiDataObjRepl in parallel (race condition?)​

​Inconsistency in the iRODS catalog, two data objects can share the same DATA_ID.​

If this subject gave you heart palpitations, as it did me, then I will quote a pertinent part of the discussion;

"This is not that two different data objects are ‘sharing the DATA_ID’, but rather, during a rename operation via webdav… one of the replicas of a single DATA_ID was left behind and didn’t get the rename treatment. They are the same data object, but with a DATA_NAME corruption/bug we may have already fixed with the work going into 4.2.9."

​irm with force flag should not care about “does not exist”​

​Replace all use of server_properties API with nlohmann::json library​

​double free with ibun to replication resource​

​Undocumented new dynamic PEPs pep_api_replica_[open|close]_[pre|post|except|finally]​

​Removing soft/sym link to an iRODS collection removes the collection, not the link​

​unixfilesystem should try harder if a stat says the file is not present​

Mostly important for spossibly transitory filesystems like NFS, FUSE etc etc, I suspect.

​irodsctl status does not print any messages to the terminal​

master and 4-2-stable… not released yet perhaps.

​irsync does not honor “ignore symlinks flag –link”, when symlink is broken​

​Remove group quota functionality​

This issue appears here because its had recent activity, and actually, Logical Quotas are the way to go for quotas in iRODS unless you have only a per-user quota and users never share quota. Otherwise, it all gets a bit messy, as this issue discussion shows.

​Add detached mode to unixfilesystem plugin​

Could be very interesting for NFS or Lustre backed “unix filesystem” based resources and HA design.

​irepl silently ignores -S in composite resource, doesn’t update replicas​

​add pre-route ping/heartbeat to resolve_resource_hierarchy​

Something for 4.3.x…

​create irodsMonitor binary​

As above

​remove iphybun​

If you care about this, remember, vote early and often here!

​delayed rule does not execute with same privs as submitter does​

​iquest fails when select is used in argument string​

​The database schema is inefficient​

​define the rebalance operation for random and roundrobin resource plugins​

​Virtual collection hierarchies​

​Add CRC as a checksum option​

​Allow in-place registering of non-unixfilesystem data objects​

​add rule priority to delayed rules​

​Behavior of imkdir -p with hidden collections​

​set delay rule quota for user​

​DNS round-robin with name service cache (especially OS X)​

​use of TCP_NODELAY in socket setup​

​features request​

​Add Column to R_DATA_MAIN for Unique Document Identifier​

​Accessing a file stored in a bundle resource from a remote zone​

​Allow clients to determine collation locale used to sort e.g. GenQuery results​

​Potential issues with collections in general queries​

​Resource context verification at resource creation​

​Code signing infrastructure for apps​

​Access to ‘cache’-type resources should be virtualized​

​Softlink for metadata​

​Path to “trash” should be parameterized​

​Database thoughts - Intersect​

​MIME type standardization​

​A grab bag of questions and previews of DFC noted client enhancements​

​Hidden files (.file) and special meaning directories​

​Tickets and io streams​

​Add secure AVU metadata types​

​Current behavior when doing recursive iputs involving symlinks is possibly surprising.​

​ils -L shows full paths - is this information leakage?​

​enforce federation exclusively between iCAT servers​

​Check permissions on AVUs​

​change ichmod to handle the full range of the iRODS permission model​

Closed Issues

Still keeping some of the entries below despite them not being closed this month but instead updatedor referenced by other issues. If it’s too noisy for you, do hit ‘reply’ and let me know what would work better?

closed on - 2021-03-08 14:46:34 writeLine with stdout not forwarding to serverLog​

closed on - 2021-03-04 12:34:12 Should iphymv respect minimum_free_space_for_create_in_bytes?​

iphymv bypasses a lot of policy as its treated as an admin level tool, and so there to get you out of jams. In this case, filling a resource despite “Minimum Free Space For Create In Bytes” being set. If thats not what you had in mind, this also contains a sample rule that allows the admin to enforce that limitation.

closed on - 2021-02-23 16:57:32 Disable Xmsg tests​

closed on - 2020-11-10 12:28:49 sending ‘PING’ to 1247 causes -4000 error in log​

closed on - 2021-03-16 18:51:01 SHA512 hasher​

closed on - 2021-02-23 18:51:59 irodsReServer logging waitAndFreeReThr and not releasing memory​

closed on - 2021-03-01 20:05:51 iput: checksum with bulk transfer​

closed on - 2021-02-23 18:10:29 4.1.8 - roundrobin resource can fill one child, leaving others empty​

closed on - 2016-02-16 18:32:42 choose a compiler for 4.2+​

closed on - 2016-08-12 20:01:06 remove perl dependency​

closed on - 2015-06-22 18:21:33 sketch of a chunked transfer​

closed on - 2016-02-16 18:27:43 Parallel put with overwrite flag will delete an existing file on failure​

closed on - 2016-08-12 19:21:18 pluggable rule engine​

closed on - 2015-06-11 16:08:34 Review and revamp WSO support​

closed on - 2015-03-27 19:18:47 propagating checksums under replication resources​

closed on - 2016-02-16 18:26:38 Add optional clean up of partial files from failed upload​

closed on - 2015-03-16 18:21:07 cannot remove file if conflicting file exists in trash​

closed on - 2015-08-19 20:30:52 Create .irodsEnv parameter for default client NumThreads​

closed on - 2016-08-03 18:32:05 create a server runtime library​

closed on - 2016-10-31 18:22:26 internal system heartbeat capability​

closed on - 2015-04-15 01:43:08 Key-Value passthrough from icommands to plugins​

closed on - 2016-02-16 18:19:09 put data with metadata as atomic operation​

closed on - 2015-06-22 17:45:28 iRODS streaming of files​

​