The Resource - November 2020
Hello Reader, here is this month's iRODS news and developments!
If you're facing an issue with iRODS you're not sure how to solve, please do drop me a line; if I've come across a solution or seen something relevant elsewhere I'll do my best to let you know.
Or just drop me a mail to say 'Hi'. Always nice to hear from people, particularly in these pandemic times!
I'd love your thoughts and feedback on how this newsletter could be better for you: https://docs.google.com/forms/d/e/1FAIpQLSfODAa7U4ST9U9Tuc6S1PlQhPtoiFybKzXVgtoVnkx7ISe41A/viewform?usp=sf_link
News
----
Dave Fellinger wrote a good overview of some of the HPC users of iRODS; https://insidehpc.com/2020/11/curating-discovering-and-disseminating-hpc-research-elements-using-irods/
If you use Lustre and iRODS you might be interested in Marcin Stolarek's recent post about integrating the two; https://funinit.wordpress.com/2020/11/13/how-irods-can-set-striping-on-lustre/amp/?__twitter_impression=true
November Development update was released; https://us8.campaign-archive.com/?e=fed5806c43&u=881549ea51417a5dfcfa8a8e7&id=e856ac1548#mctoc1
SoftIron Joins the consortium; Managed CEPH clusters looks like - https://us8.campaign-archive.com/?e=fed5806c43&u=881549ea51417a5dfcfa8a8e7&id=e856ac1548#mctoc2
The last #TRiRODS of 2020 will be held virtually on December 2. https://t.co/4CuvrHYgwr?amp=1
Globus iRODS connector is announced; https://irods.org/2020/11/globus-for-irods-connector-is-released/
New Release: #Python #iRODS Client v0.8.4 https://github.com/irods/python-irodsclient
RODS Web Application Metalnx 2.2.0 is released; https://github.com/irods-contrib/metalnx-web/releases
if you want to go straight to the Docker image: https://hub.docker.com/r/irods/metalnx/tags
Main Repository Activity
------------------------
Specific Query from rules written in iRODS Rule Language - https://github.com/irods/irods/issues/5248
As I understand this, you can write your own queries (`iadmin asq`) and then refer the the results using the inbuilt rules language.
remove simplequery - https://github.com/irods/irods/issues/5247
Anyone still using this in their deployment or rules?
irods-icommands installation fails on Ubuntu20 - https://github.com/irods/irods/issues/5246
Its not released for 20.04 yet, but the issues points at a way to compile yourself.
There is also some discussion (in another issue or chat I can't find now) around singularity images,which might be more helpful to HPC types who usually can't run Docker (although there are icommands ones of those around too).
Transfer protocol that supports concurrent server and client-side checksumming - https://github.com/irods/irods/issues/5245
Interesting feature request that might speed up large file checksumms.
iput progress output with client-side checksumming - https://github.com/irods/irods/issues/5241
Related to the above, a request to make it clearer when the icommands are generating or verifying checksums.
Failed to update irods_environment.json on brand-new iinit - https://github.com/irods/irods/issues/5239
unix vault security: mkdir before iCAT access control? - https://github.com/irods/irods/issues/5235
Fixed in 4.2.8, another reason to upgrade if you're running a version of 4.2 before that!
Oracle Database Plugin possible memory leak - https://github.com/irods/irods/issues/5234
Possibly? Anyone else have lightly used Oracle Providers suddently go boom? Just me?
iput returning zero on invalid upload - https://github.com/irods/irods/issues/5231
Still under investigation this one...
msi_update_unixfilesystem_resource_free_space being run on incorrect server? - https://github.com/irods/irods/issues/5230
Possibly just logging on the wrong system?
replication resource stacktrace - INVALID_LEXICAL_CAST - https://github.com/irods/irods/issues/5227
remote() throwing TypeError when interacting with boost::python - https://github.com/irods/irods/issues/5226
Document how to increase the max length of syslog messages - https://github.com/irods/irods/issues/5225
leftover 'temporary' rulefile.re.[pid] causing stacktraces once PIDs wrap - https://github.com/irods/irods/issues/5224
Worth clearing up your /etc/irods directory if you use rule files beyond the core.re.
Make End User errors more user friendly - https://github.com/irods/irods/issues/5221
Not every end user checks their return codes. I KNOW, right? Still....
iRODS buildsystem does not make proper use of CMAKE_INSTALL_PREFIX or GNUInstallDirs - https://github.com/irods/irods/issues/5220
I like this person...
iRODS does not follow XDG Base Directory Specification - https://github.com/irods/irods/issues/5219
See, told you I liked them...
parallel transfer in a network with firewalls drops 'long idle' connections - https://github.com/irods/irods/issues/5218
Reveals some tweaks you can do to sysctl to avoid this. Hopefully being added to the docs?
issue with iput/iget and renewing socket connection - https://github.com/irods/irods/issues/5217
Anyone else seeing issues with `-T` and large files?
Memory leak in rcSpecificQuery - https://github.com/irods/irods/issues/5216
Consolidate duplicate symbols in shared libraries - https://github.com/irods/irods/issues/5212
Yay for shorter build times and less requirements!
iadmin on a consumer should refuse to create any resource it can't discover with iadmin lt resc_type - https://github.com/irods/irods/issues/5211
irods rule language temporary rule files not temporary enough - https://github.com/irods/irods/issues/5210
Implement tests for icommand REPLs - https://github.com/irods/irods/issues/5207
Weird orphaned directory - https://github.com/irods/irods/issues/5206
iRODS storage server process crash - https://github.com/irods/irods/issues/5205
Control-character safety in filenames; -print0 option; NUL-terminated lines - https://github.com/irods/irods/issues/5202
pep_api_auth_response_pre can be triggered without _post or _except - https://github.com/irods/irods/issues/5201
iphymv (et al?) returns SYS_USER_NO_PERMISSION when it should be ENOENT (or equivalent) - https://github.com/irods/irods/issues/5177
Default number of threads advanced setting should be 3 - https://github.com/irods/irods/issues/5171
Debatable, that's optimised for 10G, but HPC systems can use dual 25G bonded, or, if rich, dual 100G.
Dont come to me with talk about this 'legacy 40G' stuff though!
Delay server updates the log level for the wrong log category - https://github.com/irods/irods/issues/5169
ilsresc -l prints CAT_NO_ROWS_FOUND with exactly 350 resources, with return code 0 - https://github.com/irods/irods/issues/5155
You'd think 350 would be enough for anyone, wouldn't you? Well....
Fortunately its 'just' a paging issue.
Document Limitations and Restrictions - https://github.com/irods/irods/issues/5073
I think this has been around a bit but got updated just recently...
Replication resource can target the wrong logical path on subsequent operations reusing the same connection - https://github.com/irods/irods/issues/5072
Awooga! This is a big one folks if you use replication composite tree's you'll want to be updating to 4.2.9 when it comes out!
imeta qu numeric comparison throws sql error - https://github.com/irods/irods/issues/5019
icp breaks when data_object name contains "' and '" - https://github.com/irods/irods/issues/4983
extend retry to all commands - https://github.com/irods/irods/issues/4976
Good discussion around the options available, but then I would say that.
icp should not be responsible for creating intermediate collections - https://github.com/irods/irods/issues/4973
iget does not continue progress output when retries invoked - https://github.com/irods/irods/issues/4968
This is an odd one. Also do a search on the repo for `with_durablility` for other ways to do retry if you're on 4.2.7 or later.
iquest cannot match names containing apostrophes via = operator - https://github.com/irods/irods/issues/4887
ilocate should support case-insensitive search, be documented - https://github.com/irods/irods/issues/4761
iquest fails when select is used in argument string - https://github.com/irods/irods/issues/4697
"iquest --sql" does not work with format string - https://github.com/irods/irods/issues/4695
The joy of quoting strikes again.
acPostProcForPut fires off also on replication - https://github.com/irods/irods/issues/4614
irepl --rlock fails with 'file name too long' when irepl by itself doesn't - https://github.com/irods/irods/issues/4433
Crash over long collection name - https://github.com/irods/irods/issues/4123
Clean up any /etc/irods/core.re.xxxx files on server stop - https://github.com/irods/irods/issues/4080
replace GenQuery ad-hoc parsing code with flex/bison parser - https://github.com/irods/irods/issues/3902
add parallel option to rebalance operation - https://github.com/irods/irods/issues/3529
Interesting discussion here.
iquest output is not Unix-friendly - https://github.com/irods/irods/issues/3135
Holding out for JSON output, personally. Hmm. Did I actually raise an request for that?
Renaming groups - https://github.com/irods/irods/issues/2446
Closed Issues
-------------
Optimized build for Travis-CI (master) - https://github.com/irods/irods/pull/5222 - closed on 2020-11-04 19:27:53
All the cool kids are doing this at the moment, its almost like they have changed their TOS or something....
Still, quicker builds is all good.
icommands iuserinfo: use after free in main - https://github.com/irods/irods/issues/5214 - closed on 2020-10-26 21:04:53
irods does not build with clang 10 - https://github.com/irods/irods/issues/5213 - closed on 2020-10-27 11:45:06
imeta does not print an error when failing to parse a path string - https://github.com/irods/irods/issues/5186 - closed on 2020-10-19 17:39:56
imeta ls[w] skips path validation for collections if detailed output is requested - https://github.com/irods/irods/issues/5185 - closed on 2020-10-19 17:40:07
imeta addw prints "AVU added to x data-objects" to stdout for valid paths regardless of success/error - https://github.com/irods/irods/issues/5184 - closed on 2020-10-19 17:39:47
izonereport does not include resc_id information - https://github.com/irods/irods/issues/5170 - closed on 2020-10-20 17:55:10
This will make scripting from the output of izonereport easier, as you couldn't build the tree from the current JSON.
imeta addw gives error code 4 but adds metadata - https://github.com/irods/irods/issues/5102 - closed on 2020-10-19 17:39:35
imeta addw returns non-zero on seeming success - https://github.com/irods/irods/issues/5101 - closed on 2020-10-19 17:39:29
imeta now exits when it encounters an empty line on STDIN, rather than ^D - https://github.com/irods/irods/issues/5081 - closed on 2020-10-22 18:26:23
imeta qu should return non-zero on Unrecognised input - https://github.com/irods/irods/issues/5021 - closed on 2020-10-19 17:39:21
sending 'PING' to 1247 causes -4000 error in log - https://github.com/irods/irods/issues/3880 - closed on 2020-11-10 12:28:49
Good this is documented now.
iquest (4.1.9) How to get RESC_LOC for a file in a compound resource? - https://github.com/irods/irods/issues/3288 - closed on 2020-11-12 00:06:05
irodsReServer logging UNIX_FILE_STAT_ERR - https://github.com/irods/irods/issues/3286 - closed on 2016-09-19 12:51:02
irm, irmtrash don't delete physical folders - https://github.com/irods/irods/issues/3124 - closed on 2020-11-06 20:16:01
Need to document the maximum host name length of 64 characters - https://github.com/irods/irods/issues/2777 - closed on 2020-10-21 19:23:39
---------------------------------------------------------------------------------------------------------
If you think someone else would appreciate this newsletter, they can sign up at https://theresource.metadata.school/
7 Yaks were shaved in the making of this newsletter.