The Resource for October 2022
Hello Reader, here is this month’s iRODS news and developments!
News
RENCI at SuperComputing 2022 Conference (Dallas, Texas, US)
They will be holding a mini workshop Protocol Plumbing: Presenting iRODS as WebDAV, FUSE, REST, NFS, SFTP, K8s CSI, and S3
iRODS Python iRODS Client v1.1.5 is released
1.1.5
- [#383] correct logical path normalization
- [#369] remove dynamic generation of message classes
- [#386][#389] only load timestamps when requested
- [#386] initial change to add create and modify times for metadata
iRODS Zone Management Tool 0.3.0 is released
0.3.0
Updated to require C++ REST API version 0.9.2 or later
Development Update: October 2022
Development Update, October 2022
Main Repository Activity
Open Issues
Remove property from log message output
The logger in iRODS hard-codes the value of log_facility to local0. This property and value are not important to the operation of iRODS and should be removed.
Add zone name to log message output
See above!
Oddity in when passed invalid parameters for resource type creation
iadmin mkresc doesn’t validate its strings completely.
Refactor client_api_allowlist interface to match style of replica_access_table, etc
While the client api allowlist's functionality will remain the same, we should change how it is held to match other modern iRODS singleton interfaces.
This is more about interface consistency than anything.
Deprecate SimpleQuery
See #6624 for more details about how we got here...
Which was due to iadmin lu having some odd behaviour on username length.
Remove all Microsoft Windows specific code
Not been used for a while.
Investigate main server process memory leak
iRODS errors codes attached or pointing to their documentation
Okay, so the tasks here are:
Add mention/description of LOCKED_DATA_OBJECT_ACCESS and INTERMEDIATE_REPLICA_ACCESS in Troubleshooting for "Data object stuck in locked (or intermediate) status"
Add new page for columns available to iquest (aka iquest attrs)
Hurrah, more docs!
Microservice support for compressed structured files.
Feature Request;
ibun supports compressed structured files, but I have found no microservices that similarly do so. All I see is msiTarFileCreate/Extract and they don't support compression. Creating a generic structured file microservice pair that supports compression and data type is one idea, another would be to have a separate microservice pair for each data type supported by ibun.
Add a ‘force’ option to msiTarFileExtract
msiTarFileCreate contains a force switch, but msiTarFileExtract does not. It would be nice to have similar options for complimentary microservices. Also, some microservices use a dedicated parameter for this while others include it in a msKeyValStr. And the process status is sometimes returned both directly and by reference (msiTarFileExtract) and sometimes only directly (msiTarFileCreate). Consistency in these would also be nice.
iadmin fails to list user in case of particular username length combinations
See ‘deprecate SimpleQuery’, above
itree -P --pattern seems not working correctly
I learned more about than I had known before I read this issue. If you care about using regex’s with then this issue is for you!
itree not handling wrong args for options properly
closeAllL1desc should not call PEPs
msiExecCmd sets up communication pipes, forks, closes excess file descriptors and executes a program.
The file descriptors are closed by closeAllL1desc, which is called from nowhere else. It starts at L1 to discover the descriptors, and then goes back up to L3 to close them. However, this has all sorts of effects -- apart from closing the file descriptors -- that are unwanted at this point. It is not appropriate that pep_resource_close_pre is called because the relevant resource is still open in the parent process, and it's not appropriate to run any rules in a forked process.
Refactor resource administration API to throw exceptions instead of return error codes
Add C++ library for managing zones
Automate use of sanitizers (address, thread, undefined behaviour, etc.)
Possible segmentation fault.
Consider dropping support for RBUDP
Does anyone use RBUDP for data movement?
If no one relies on it, we should remove it.
Not much support for the protocol so far, so it is in your workflow, say now!
Main server process does not honour log level changes following irods-grid --all reload
quota bug in v4.3
Seems to be a bug in reducing quota in 4.3
imkdir idempotency handling
Ubuntu 22.04 Version Support
We will support Ubuntu22.04 in an upcoming 4.3.x release. No known blockers at this time.
We will not support Ubuntu22.04 for 4.2.x. - see earlier discussion at #4883.
Should iRODS enforce always having one OWN permission on collections and data objects
Related; “ichmod should not be allowed to bypass the permission model #6579”
Manipulating resources and hierarchies is not reflected in the resc_mgr
For long running agents, updating one or more resources means the resc_mgr needs to be updated as well. Not updating the resc_mgr leads to incorrect results. It also means iRODS may not be able to carry out certain operations because the catalog and resc_mgr are not synchronized.
Closed Issues
Closed on - 2022-10-07 22:03:02 iscan documentation implies that it accepts multiple paths as command line arguments
Spoiler; it doesn’t.
Closed on - 2022-09-28 12:55:36 iquest: problems with numeric relops
Request for further documentation on n> and similar operations in iquest
The only documentation I see that speaks of those operators is here: https://docs.irods.org/4.3.0/icommands/metadata/#qu
We'll look into providing more documentation for this.
As for new operators, there are no plans to add new functionality to the existing GenQuery implementation. The new GenQuery parser we're working on will address these issues.
Closed on - 2022-10-04 19:16:52 Remove setup logic for rsyslog and logrotate
targetted for 4.3.1, with the new syslog logging introduced in 4.3
iRODS does not declare any dependencies on rsyslog or logrotate, yet setup includes logic that attempts to configure and launch them.
Given that there are multiple implementations of syslog, iRODS cannot assume all deployments will use rsyslog. Choosing which syslog implementation to use must be made by the administrator.
Therefore, all logic focused on setting up these services should be removed. Documentation should be provided that helps administrators with configuring syslog and logrotate.
Really nice updated Documentation with an example for rsyslog.
Closed on - 2022-09-27 14:22:37 DataObjectManager.open promoting mode ‘w’ to mode ‘w+’ when opening existing data object
Note that it appears to be server behaviour, not the python client behaviour.
I'm using python-irodsclient version 1.1.4 with iRODS 4.2.10. I use irods.manager.DataObjectManager.open to open a data object using open mode w and implicitly allow the data object to be created, If the data object already exists, the library appears to change the open mode to w+. This changes the access to the data object from write-only to read-write.
Closed on - 2022-10-15 13:12:44 Clang-Tidy: Consider disabling readability-function-cognitive-complexity
Closed on - 2022-09-27 19:49:38 Investigate test failures in test_configuration_reload.py
Closed on - 2022-09-19 14:03:34 Is these Iget and Iput speed correct?
I've verified that the math is correct. We suspect the speeds reported by you are because the transfers are all local (no remote network involved) and the OS is being very smart.
Linux caching FTW! A good summary if you’ve not come across this
Closed on - 2022-09-26 17:05:34 Adding a duplicate AVU should result in an error
4.3.0 - fixed in 4.3.1
Closed on - 2022-10-11 14:19:56 Update documentation regarding replica statuses
An excellent update documenting how the replica statuses work. Will appear in 4.2.11 (and presumably 4.3.1)
Closed on - 2022-10-04 19:16:41 Fix non-package installations for 4.3.1
Closed on - 2022-10-10 16:51:36 ifsck help implies multiple paths on command line allowed
Spoiler; they aren’t.
Closed on - 2022-10-03 22:35:29 Design/Implement new C++ wrapper around tickets APIs
Closed on - 2022-09-27 19:33:11 allow user to select rule engine instance in remote execution in the rule engine
Fixed in 6465 to appear in 4.2.12 and presumably 4.3.1
Python iRODS Client Activity
Open Issues
put() ignoring minimum_free_space_for_create_in_bytes and resc_status
iRODSSession.data_objects.put() doesn't refresh the state of the catalogue after/before each call.
This means that the only way to ensure a clean catalogue each time is to have the put() happen after a session = iRODSSession and immediately followed by a session.cleanup(). It’s inefficient but arguably no worse than an iput, however, the README examples and documentation need to state this explicitly.
(disclaimer; quoting myself…)
Catching and translating iRODS errors
Unfortunately put( ...) in PRC isn't quite the same as iput - one difference being it doesn't use the PUT api
and after some more digging it was found that
Looking at this again, all we get as repr( )-style output is UNIX_FILE_WRITE_ERR(None,), and ....
We could have at least included the errno code (which was 28) and/or the symbol and strerror-message [ (ENOSPC) / 'No space left on device']. I'll make this an issue in its own right.
This led to #400 Expose errno code in string representation of an irods.exception
Group based permission works differently for collections
Setting a group permission on a collection exposes all users of the group. However this works normal for data objects.
and
This can be closed because I have learned that the keyword argument report_raw_acls is controlling that. Though I guess this comes already as True for data objects by default.
Getting permission does not bring g: prefix for groups
Querying acl on (getting permission of ) an object doesn't bring g prefix specifying group based permission although there is already an granted group permission exists. AccessManager behaves it like a normal user. On the other hand ils -A shows it is a group user.
Closed Issues
Closed on - 2022-09-20 15:41:16 omitting admin flag as obj.metadata() arg causes error
Closed on - 2022-09-20 15:41:06 Expose create and modify times for metadata
Allthough metadata items can be queried and filtered for create_time/modify_time columns, the higher level class iRODSMeta does not expose these fields. Consequently, they are not available when fetching metadata from collections and data objects (and resources and users).
Closed on - 2022-09-21 04:10:25 [On Windows, path normalization replaces “/” with “"](https://github.com/irods/python-irodsclient/
issues/383)
get( ) and other methods call logical path normalization implicitly, so this is a priority to fix.
the irods.path.iRODSPath carelessly uses os.normpath which on MSWindows applies the finishing touch of replacing any forward slashes ( / ) with the backward ones ( \ ) more standard on that OS.
Closed on - 2022-09-20 19:05:57 remove dynamic class reconfiguration
packstruct tags which have been added to an API can be left in during client communications with older servers and they will be ignored (affects metadata , ticket APIs)
NFSRODS Activity
Closed Issues
Closed on - 2022-10-12 17:24:41 Building nfsrods in an automation fails
Docker command had a left in interactive flag (-it)
icommands Activity
Closed Issues
Closed on - 2022-09-19 19:17:34 Bash autocomplete for iCommands not working right
Lead to add bash and tcsh completion scripts to this repository #30 in irods/contrib (which I’ve not been watching - should I? Let me know if you would want updates to that repository reflected in this email).
YODA Activity
Open Issues
[FEATURE] Affiliation identifier field
The affiliation field in the metadata form is a free text form that invites different types of spelling of affiliation (such as UU, Utrecht University; or WUR, Wageningen UR, Wageningen University and Research, Wageningen University & Research, Wageningen University, etc). Although a person identifier can be provided, an affiliation identifier is not present.
Closed Issues
Closed on - 2022-10-12 06:38:42 [BUG] missing script admin-tape-archive-set-state.sh in py_acPreProcForExecCmd
Closed on - 2022-10-12 06:58:06 [BUG] Ansible does not pull the latest scripts from GitHub during the deployment
During the deployment with ansible we noticed that the scripts located in /var/lib/irods/msiExecCmd_bin were not the latest ones available in the Yoda github. For example "admin-tape-archive-set-state.sh".
Closed on - 2022-09-23 07:35:45 Menu always to foreground
Closed on - 2022-09-20 15:00:45 [FEATURE] For EUA make ‘domain’ variable
Please make the user's 'domain' variable instead of uu.nl.
If you think someone else would appreciate this newsletter, they can sign up at https://theresource.metadata.school/
No Yaks were shaved in the making of this newsletter. I probably should go looking for some.