The Resource logo

The Resource

Archives
Subscribe
October 28, 2022

The Resource for October 2022

Hello Reader, here is this month’s iRODS news and developments!

News

RENCI at SuperComputing 2022 Conference (Dallas, Texas, US)

They will be holding a mini workshop Protocol Plumbing: Presenting iRODS as WebDAV, FUSE, REST, NFS, SFTP, K8s CSI, and S3​

iRODS Python iRODS Client v1.1.5 is released

​1.1.5​

  • [#383] correct logical path normalization
  • [#369] remove dynamic generation of message classes
  • [#386][#389] only load timestamps when requested
  • [#386] initial change to add create and modify times for metadata

​Changes from 1.1.4​

iRODS Zone Management Tool 0.3.0 is released

​0.3.0​

Updated to require C++ REST API version 0.9.2 or later

​Changes from 0.2.0​

Development Update: October 2022

​Development Update, October 2022​

Main Repository Activity

Open Issues

​Remove property from log message output​

The logger in iRODS hard-codes the value of log_facility to local0. This property and value are not important to the operation of iRODS and should be removed.

​Add zone name to log message output​

See above!

​Oddity in when passed invalid parameters for resource type creation​

iadmin mkresc doesn’t validate its strings completely.

​Refactor client_api_allowlist interface to match style of replica_access_table, etc​

While the client api allowlist's functionality will remain the same, we should change how it is held to match other modern iRODS singleton interfaces.
This is more about interface consistency than anything.

​Deprecate SimpleQuery​

See #6624 for more details about how we got here...

Which was due to iadmin lu having some odd behaviour on username length.

​Remove all Microsoft Windows specific code​

Not been used for a while.

​Investigate main server process memory leak​

​iRODS errors codes attached or pointing to their documentation​

Okay, so the tasks here are:
Add mention/description of LOCKED_DATA_OBJECT_ACCESS and INTERMEDIATE_REPLICA_ACCESS in Troubleshooting for "Data object stuck in locked (or intermediate) status"
Add new page for columns available to iquest (aka iquest attrs)

Hurrah, more docs!

​Microservice support for compressed structured files.​

Feature Request;

ibun supports compressed structured files, but I have found no microservices that similarly do so. All I see is msiTarFileCreate/Extract and they don't support compression. Creating a generic structured file microservice pair that supports compression and data type is one idea, another would be to have a separate microservice pair for each data type supported by ibun.

​Add a ‘force’ option to msiTarFileExtract​

msiTarFileCreate contains a force switch, but msiTarFileExtract does not. It would be nice to have similar options for complimentary microservices. Also, some microservices use a dedicated parameter for this while others include it in a msKeyValStr. And the process status is sometimes returned both directly and by reference (msiTarFileExtract) and sometimes only directly (msiTarFileCreate). Consistency in these would also be nice.

​iadmin fails to list user in case of particular username length combinations​

See ‘deprecate SimpleQuery’, above

​itree -P --pattern seems not working correctly​

I learned more about than I had known before I read this issue. If you care about using regex’s with then this issue is for you!

​itree not handling wrong args for options properly​

​closeAllL1desc should not call PEPs​

msiExecCmd sets up communication pipes, forks, closes excess file descriptors and executes a program.
The file descriptors are closed by closeAllL1desc, which is called from nowhere else. It starts at L1 to discover the descriptors, and then goes back up to L3 to close them. However, this has all sorts of effects -- apart from closing the file descriptors -- that are unwanted at this point. It is not appropriate that pep_resource_close_pre is called because the relevant resource is still open in the parent process, and it's not appropriate to run any rules in a forked process.

​Refactor resource administration API to throw exceptions instead of return error codes​

​Add C++ library for managing zones​

​Automate use of sanitizers (address, thread, undefined behaviour, etc.)​

​Possible segmentation fault.​

​Consider dropping support for RBUDP​

Does anyone use RBUDP for data movement?
If no one relies on it, we should remove it.

Not much support for the protocol so far, so it is in your workflow, say now!

​Main server process does not honour log level changes following irods-grid --all reload​

​quota bug in v4.3​

Seems to be a bug in reducing quota in 4.3

​imkdir idempotency handling​

​Ubuntu 22.04 Version Support​

We will support Ubuntu22.04 in an upcoming 4.3.x release. No known blockers at this time.
We will not support Ubuntu22.04 for 4.2.x. - see earlier discussion at #4883.

​Should iRODS enforce always having one OWN permission on collections and data objects ​

Related; “ichmod should not be allowed to bypass the permission model #6579”

​Manipulating resources and hierarchies is not reflected in the resc_mgr​

For long running agents, updating one or more resources means the resc_mgr needs to be updated as well. Not updating the resc_mgr leads to incorrect results. It also means iRODS may not be able to carry out certain operations because the catalog and resc_mgr are not synchronized.

Closed Issues

Closed on - 2022-10-07 22:03:02 iscan documentation implies that it accepts multiple paths as command line arguments​

Spoiler; it doesn’t.

Closed on - 2022-09-28 12:55:36 iquest: problems with numeric relops​

Request for further documentation on n> and similar operations in iquest

The only documentation I see that speaks of those operators is here: https://docs.irods.org/4.3.0/icommands/metadata/#qu
We'll look into providing more documentation for this.
As for new operators, there are no plans to add new functionality to the existing GenQuery implementation. The new GenQuery parser we're working on will address these issues.

Closed on - 2022-10-04 19:16:52 Remove setup logic for rsyslog and logrotate​

targetted for 4.3.1, with the new syslog logging introduced in 4.3

iRODS does not declare any dependencies on rsyslog or logrotate, yet setup includes logic that attempts to configure and launch them.
Given that there are multiple implementations of syslog, iRODS cannot assume all deployments will use rsyslog. Choosing which syslog implementation to use must be made by the administrator.
Therefore, all logic focused on setting up these services should be removed. Documentation should be provided that helps administrators with configuring syslog and logrotate.

Really nice updated Documentation with an example for rsyslog.

Closed on - 2022-09-27 14:22:37 DataObjectManager.open promoting mode ‘w’ to mode ‘w+’ when opening existing data object​

Note that it appears to be server behaviour, not the python client behaviour.

I'm using python-irodsclient version 1.1.4 with iRODS 4.2.10. I use irods.manager.DataObjectManager.open to open a data object using open mode w and implicitly allow the data object to be created, If the data object already exists, the library appears to change the open mode to w+. This changes the access to the data object from write-only to read-write.

Closed on - 2022-10-15 13:12:44 Clang-Tidy: Consider disabling readability-function-cognitive-complexity​

Closed on - 2022-09-27 19:49:38 Investigate test failures in test_configuration_reload.py​

Closed on - 2022-09-19 14:03:34 Is these Iget and Iput speed correct?​

I've verified that the math is correct. We suspect the speeds reported by you are because the transfers are all local (no remote network involved) and the OS is being very smart.

Linux caching FTW! A good summary if you’ve not come across this​

Closed on - 2022-09-26 17:05:34 Adding a duplicate AVU should result in an error​

4.3.0 - fixed in 4.3.1

Closed on - 2022-10-11 14:19:56 Update documentation regarding replica statuses​

An excellent update documenting how the replica statuses work. Will appear in 4.2.11 (and presumably 4.3.1)

Closed on - 2022-10-04 19:16:41 Fix non-package installations for 4.3.1​

Closed on - 2022-10-10 16:51:36 ifsck help implies multiple paths on command line allowed​

Spoiler; they aren’t.

Closed on - 2022-10-03 22:35:29 Design/Implement new C++ wrapper around tickets APIs​

Closed on - 2022-09-27 19:33:11 allow user to select rule engine instance in remote execution in the rule engine​

Fixed in 6465 to appear in 4.2.12 and presumably 4.3.1

Python iRODS Client Activity

Open Issues

​put() ignoring minimum_free_space_for_create_in_bytes and resc_status​

iRODSSession.data_objects.put() doesn't refresh the state of the catalogue after/before each call.

This means that the only way to ensure a clean catalogue each time is to have the put() happen after a session = iRODSSession and immediately followed by a session.cleanup(). It’s inefficient but arguably no worse than an iput, however, the README examples and documentation need to state this explicitly.

(disclaimer; quoting myself…)

​Catching and translating iRODS errors​

Unfortunately put( ...) in PRC isn't quite the same as iput - one difference being it doesn't use the PUT api

and after some more digging it was found that

Looking at this again, all we get as repr( )-style output is UNIX_FILE_WRITE_ERR(None,), and ....
We could have at least included the errno code (which was 28) and/or the symbol and strerror-message [ (ENOSPC) / 'No space left on device']. I'll make this an issue in its own right.

This led to #400 Expose errno code in string representation of an irods.exception​

​Group based permission works differently for collections​

Setting a group permission on a collection exposes all users of the group. However this works normal for data objects.

and

This can be closed because I have learned that the keyword argument report_raw_acls is controlling that. Though I guess this comes already as True for data objects by default.

​Getting permission does not bring g: prefix for groups​

Querying acl on (getting permission of ) an object doesn't bring g prefix specifying group based permission although there is already an granted group permission exists. AccessManager behaves it like a normal user. On the other hand ils -A shows it is a group user.

Closed Issues

Closed on - 2022-09-20 15:41:16 omitting admin flag as obj.metadata() arg causes error​

Closed on - 2022-09-20 15:41:06 Expose create and modify times for metadata​

Allthough metadata items can be queried and filtered for create_time/modify_time columns, the higher level class iRODSMeta does not expose these fields. Consequently, they are not available when fetching metadata from collections and data objects (and resources and users).

Closed on - 2022-09-21 04:10:25 [On Windows, path normalization replaces “/” with “"](https://github.com/irods/python-irodsclient/

issues/383)

get( ) and other methods call logical path normalization implicitly, so this is a priority to fix.
the irods.path.iRODSPath carelessly uses os.normpath which on MSWindows applies the finishing touch of replacing any forward slashes ( / ) with the backward ones ( \ ) more standard on that OS.

Closed on - 2022-09-20 19:05:57 remove dynamic class reconfiguration​

packstruct tags which have been added to an API can be left in during client communications with older servers and they will be ignored (affects metadata , ticket APIs)

NFSRODS Activity

Closed Issues

Closed on - 2022-10-12 17:24:41 Building nfsrods in an automation fails​

Docker command had a left in interactive flag (-it)

icommands Activity

Closed Issues

Closed on - 2022-09-19 19:17:34 Bash autocomplete for iCommands not working right​

Lead to add bash and tcsh completion scripts to this repository #30 in irods/contrib (which I’ve not been watching - should I? Let me know if you would want updates to that repository reflected in this email).

YODA Activity

Open Issues

​[FEATURE] Affiliation identifier field​

The affiliation field in the metadata form is a free text form that invites different types of spelling of affiliation (such as UU, Utrecht University; or WUR, Wageningen UR, Wageningen University and Research, Wageningen University & Research, Wageningen University, etc). Although a person identifier can be provided, an affiliation identifier is not present.

Closed Issues

Closed on - 2022-10-12 06:38:42 [BUG] missing script admin-tape-archive-set-state.sh in py_acPreProcForExecCmd​

Closed on - 2022-10-12 06:58:06 [BUG] Ansible does not pull the latest scripts from GitHub during the deployment​

During the deployment with ansible we noticed that the scripts located in /var/lib/irods/msiExecCmd_bin were not the latest ones available in the Yoda github. For example "admin-tape-archive-set-state.sh".

Closed on - 2022-09-23 07:35:45 Menu always to foreground​

Closed on - 2022-09-20 15:00:45 [FEATURE] For EUA make ‘domain’ variable​

Please make the user's 'domain' variable instead of uu.nl.

If you think someone else would appreciate this newsletter, they can sign up at https://theresource.metadata.school/​

No Yaks were shaved in the making of this newsletter. I probably should go looking for some.

Don't miss what's next. Subscribe to The Resource:
Powered by Buttondown, the easiest way to start and grow your newsletter.