The Resource logo

The Resource

Archives
Subscribe
August 25, 2022

August 2022 Edition of The Resource

Hello Reader, here is this month’s iRODS news and developments!

If you’re facing an issue with iRODS you’re not sure how to solve, please do drop me a line; if I’ve come across a solution or seen something relevant elsewhere, I’ll do my best to let you know. Or just drop me a mail to say ‘Hi’. Always nice to hear from people, particularly in these pandemic times!

I’d love your thoughts and feedback on how this newsletter could be better for you.

News

2022 UGM Videos are live!

​35 talks total​

Abstracts and slides at https://irods.org/ugm2022

TRiRODS Aug 2022 - Interns!

​Refactor and modernize the main irodsServer - June Releford​

Might also cover;

  • Refactor and modernize the main irodsServer
  • Add irodsServer configuration live reload
  • AMQP Audit Rule Engine Plugin Investigation and iRODS C++ REST API Additions
  • New Ticket Library
  • iRODS Testing Environment Web Application

(I confess, I have not yet watched it)

RENCI looking for their next C++ Core Developer

Required: 4y degree, modern C++, a love of open source

Unofficial iRODS Docs are starting to take shape!

As mentioned in my lighting talk (not see it? Check out above!), a few people have started submitting documentation to the unofficial iRODS Documentation Project - you can too!

It being the holidays, I’ve not had as much time to contribute as I’d like, but it’s already got a few pages and three contributors!

Main Repository Activity

Open Issues

​Remove pyparsing python module​

​Add query functions to new ticket administration library​

This would give developers a way to query info about tickets. 

​Add existence check functions to new ticket administration library​

​Remote configuration reloading​

The configuration reload feature doesn't cover any use cases with multiple instances.
Given that, it is desirable to add this to irods-grid so that it may be invoked across the zone, something like `irods-grid reload all`

​Davrods and Dynamic Policy Enforcement Points​

Davrods does not hit the actual put API, but performs an open/write/close. Please see this comment for details on which dynamic PEPs are engaged in the put operation: UtrechtUniversity/davrods#18 (comment)
The dynamic PEPs respond to actual API calls whereas the static PEPs are placed wherever the developer feels like it. In this case, the "oprType" can be set and rsDataObjClose understands this as having been a "put" and so the static PEP fires in a hard-coded location.

​Adjust clang-format and clang-tidy configuration​

​Add option that instructs the server to track the time spent in rules​

Providing such a feature would give administrators a way to see where time is spent regarding their policy and make informed decisions about how to improve it.

​iRODS rule language KeyValuePair cannot have ‘size’ has key​

​Server side implementation of Metadata library not available in RodsAPIs​

​Sporadic test failures: prepareServerSharedMemory errors on server start sometimes​

​Add markdown file that documents how to compile iRODS from source​

The iRODS repo does not provide any guidance on how to compile iRODS from source.
The project needs to introduce a new markdown file that explains how to compile iRODS. The markdown file should cover the following:
    Dependencies
    CMake options (e.g. CMAKE_INSTALL_PREFIX, CMAKE_BUILD_WITH_INSTALL_RPATH)
    iRODS options (e.g. IRODS_UNIT_TESTS_BUILD)
    Normal build steps (i.e. cmake . && ninja package)
    Recommended way of compiling iRODS (i.e. irods_development_environment)
The top-level README file can point to this new markdown file.

Hurrah! This saves me from having to put something together for the Unofficial guide… :-)

​add version number to icommands spOption (for use via ips)​

The client string printed by ips does not currently include any version information.

Alas, spOption doesn’t appear to be documented (I swear it used to be?), so here’s the source code.

​Remove .gitmodules​

​DataObjRename fails if similar named object exist in source collection​

The definitely comes under the ‘well I learned something today’;

if you do a move and rename at the same time, for example,
'imv file1 coll1/file2', it will normally succeed if there's no conflicting
data-object name in the source collection (file2) but fail (giving error
CAT_NAME_EXISTS_AS_DATAOBJ) if there is, since, internally IRODS is doing
a rename and then a move. Please handle this by running multiple separate
'imv' commands.

​Add more tests for METADATA_INCLUDED_KW(i.e. iput --metadata)​

​Merge filesystem.tpp with filesystem.hpp​

​Test downgrade of rodsadmin service account on a remote server​

​Adding a duplicate AVU should result in an error​

This appears to be a behaviour change in 4.3;

Doing this will create a problem for iput --metadata. See #6409 for details.
This change in behavior is not good for existing deployments that rely on duplicate AVUs producing an error.

​Replica in cache marked good after stage-to-cache returns an error​

This relates to the S3 plugin and Glacier.

​Tickets, Permissions, and Sharing data - A question about expectations​

It's not uncommon to give another user access to data they don't own. There are two ways to do this:
Create a ticket for the target object and give the ticket to the user
Or, modify the permissions on the target object and then on the parent collection
The point of interest here is that using a ticket doesn't require modifying permissions on the parent collection. Great. The user can't look around the parent collection. All they can do is access the data associated with the ticket.
Q. Should modifying the permissions on the parent collection be necessary?
Q. Should modifying permissions on the target object result in the same behavior as using a ticket?

​Add -f option to imv in order to allow overwrites​

The rename API currently allows overwriting when FORCE_FLAG_KW is passed via the condInput of the dataObjInp_t.
imv does not expose a way to use this feature. We should add a -f option to allow for overwrites as the API already allows for.

​icp cannot provide source resource or replica number​

icp has the ability to specify a destination resource via the -R option. It does not have an option to specify a source resource or replica number for the source data object. For many iCommands, this is -S for source resource and -n for replica number.
As with other iCommands, -S and -n should be considered incompatible parameters.
The recursive flag -r should also be considered while implementing this feature: What happens if the specified source resource or replica number is not valid for every data object in a collection?

​-R option in icp should be a directive, not a preference​

Definitely in 4.3.0, likely earlier.

Observed behavior (including steps to reproduce, if applicable)
The resource specified by -R is ignored and the existing replica on ufs0 is overwritten:

​Severe imkdir performance impact with inheritance enabled​

Still under investigation. Adding an index for the columns ACL section may help, will report back.

​Merge irods_logger.tpp with irods_logger.hpp​

​Add GitHub workflow for clang-tidy​

​Investigate how Coroutines can improve iRODS​

​test_parallel_transfer_engine on branch with apparently unrelated changes.​

​Replace Packstruct binary encoding (Pluggable Message Encoders/Decoders)​

Investigation / Enhancement
The goal of this effort is to provide a binary encoding that is maintainable, efficient, extensible, versioned, documentable, and accessible to multiple programming languages.
We can achieve this by introducing pluggable binary encoders and decoders.
Potential Serialization Libraries
    Flatbuffers
    Cap'n Proto
    Protocol Buffers
    Cereal
    MessagePack
    Simple Binary Encoding (SBE)
Things to keep in mind
    Backwards compatibility
    Federation
    Encoding negotiation

​Investigate memory usage around rsGenQuery() and large database result sets​

There is a possibility that this comes from the use of SpecficQueries (i.e. ones written by an admin and inserted with iadmin asq). The investigation continues!

​Provide mapping between iRODS error codes and POSIX error codes​

This is about categorizing iRODS error codes so that icommands and other clients return meaningful POSIX error codes instead of a random integer between 1 and 256.
This can be achieved using a map and a function. Bonus points for allowing other programming languages (e.g. Java, Python, Go, etc.) to consume the existing C/C++ mappings in a convenient manner.
Reference material:
https://man7.org/linux/man-pages/man3/errno.3.html

​Investigate specific API endpoint for groupadmin operations​

​Investigate use of liburing​

​Delayed rules do not honor remote user ownership​

= 4.2.11
The problem seems to be this: As of 579d0c2, delayed rule deletion is now correctly checking the permission level of the client user rather than the proxy user. The proxy user is always (or almost always) the service account rodsadmin account, which means that user will have permission to remove the delayed rule. Now that we are checking the client user, the usernames don't match because remote users always have the zone name appended but the delayed rule information is dropping the zone name from the username.
The solution, I think, is to write down the user name with the zone name in the delayed rule information in the catalog, and adjust the username check to account for this.

​Rocky 9 released on July 14th​

​Unattended Install non-default irods home results in error​

iRODS does not support changing the home collection during setup at this time. The setup script assumes certain things will be configured a particular way. This allows us to make assertions regarding correctness. After setup completes, you're free to change the home collection. To do that, you'd do the following:
    Create a new collection using imkdir.
    Open /var/lib/irods/.irods/irods_environment.json and set irods_home to the absolute (logical) path of the new collection.
With that said, allowing this in the future is a possibility.
Q. Is this something you need?
Q. Can you explain the motivation behind this?

​itouch non existent file creates undeletable object​

this is a known issue with itouch on the S3 resource plugin. We will look into this.

​Add safe version of irods::at_scope_exit​

​iCommands do not disconnect on SIGINT​

iCommands should handle SIGINT and disconnect from the server before aborting, if possible. This will prevent connections from staying open unnecessarily after the user has aborted the iCommand.
To be clear, connections are cleaned up by the server after the TCP connection times out, but this just handles the client-initiated terminations more gracefully.

Closed Issues

Closed on - 2022-08-16 13:54:55 Should that inheritance only affects files added to the directory after the inheritance bit is set?​

Apparently this too is where I learned something - behaviour as documented.

Closed on - 2022-08-05 23:40:08 Remove problematic resources from mkresc​

Misleading title, in my opinion - the reporter has somehow managed to make a resource with the same collection name as the homedir of the irods user. I wonder if that’s checked for in the iadmin mkresc - probably not, because someone might want to, because reasons…

Closed on - 2022-08-02 19:51:06 UseLibCXX: CMake version and LINK_LANGUAGE generator expression​

Closed on - 2022-07-29 16:30:00 Server thread policy about threads​

If you are interested in the interplay between the assorted configuration values that can determine the number of threads parallel transfer uses, this is a good issue to read through.

Closed on - 2022-08-02 19:41:33 Problem Linking CXX shared library libirods_common.so​

Closed on - 2022-07-20 13:27:00 PAM auth does not work after upgrading to 4.3.0-1​

Documentation is updated to address;

If you're using icommands, replace the use of "PAM" in your irods_environment.json file with "pam_password".

Gnnng. I’ll need to adjust all several hundred of our users configs when we upgrade. I hope the PR attached updated the docs for 4.3.0, but I suspect it will only be reflected in 4.3.1.

Closed on - 2022-07-05 15:13:51 Caught exception in migrate_delay_server​

I think this means that the amqp plugin needs bumping to 4.3.0.1?

Further investigation has revealed that this only happens with 4.3.0.0 of the audit rule engine plugin installed and configured according to the training slide deck.
As this does not happen with a default configuration or with packages built from tip of main, I think this is actually an issue with the audit rule plugin and was fixed with irods/irods_rule_engine_plugin_audit_amqp#99.

Closed on - 2022-08-10 15:45:20 Set RUNPATH instead of RPATH​

Python iRODS Client Activity

Open Issues

​Nonexistent data object raises CollectionDoesNotExist​

I think a decent summary of this issue is

PRC has historically done a query to see if the collection component of the requested new data object exists, and of course the Genquery is unable to distinguish between a collection it cannot write and one that doesn't exist.

​Uploading data raises a KeyError -1205000 once the logical quota starts monitoring ​

4.2.10/11 reported. Seems to be an issue just with the Python client and logical quotas, as icommands reported running fine.

​Large put() over federation leaves “valid” replicas of incorrect size and checksum when interrupted​

This is actually illustrating an issue with < 4.2.9, rather than a Python client issue.

​Large put() times-out with NetworkException: Could not receive server response​

Large files (here tested with >= 50 GB) timing out when uploading to a replica resource based tree. Looks like some timeouts in the Python client somewhere rather than iRODS itself…

​Retrieving large files causes unacceptable resource usage on catalog server​

This is odd, and worrying if you are trying to use the Python iRODS Client for production work without having some kind of load balancing/throttling. Seems odd behaviour compared to the normal iput/iget but perhaps I am missing something?

During a large PRC put() both catalog and resource servers have normal (< 5% increase in both CPU and RAM) resource usage, but during a large PRC get() the catalog server's resource usage spikes strongly (nearly 100% CPU and RAM/swap) while the resource server's usage remains low. The spike is quicker with more threads and the transfer rate for get() is < 20% that of put() for the same data.

​Add resource server redirect support to iput()/iget()​

… and the prior issue makes more sense now; instead of passing off the data transfer to the Consumers, it’s funnelled through the Consumer that the client connected to instead - a behaviour that usually only happens with single threaded iget when using icommands.

NFSRODS Activity

Open Issues

​access via nfsrods not working well.​

The short version of this is that NFSRODS maps NFS UID’s to the local zone users, which doesn’t work well when there isn’t a corresponding one, but there is a Federated one of the same name.

​Building nfsrods in an automation fails​

Complete with PR and discussion of fixes.

icommands Activity

Open Issues

​icp cannot copy files larger than 2 GB​

Seems to be a regression in 4.3.0 and 4.2.11 icommands talking to a 4.2.8 Zone.

As 4.2.9 made a number of changes (it really should have been 4.3.0, but I can understand not wanting to push what is now 4.3.0 to be 4.4.0!), it sounds like Zones later than 4.2.8 need to be talked to with a similar client and vice versa.

Closed Issues

Closed on - 2022-08-05 11:10:39 iCommands 4.3.0 fails to run with PAM​

The configuration value for enabling PAM in 4.3.0 has changed from "PAM" to "pam_password".

The docs are updated for 4.3.1.

Externals Activity

Open Issues

​python3-pyodbc needed in EL7​

EL7 (Centos 7, RHEL 7, and friends) needs a python3-pyodbc package. All other distros we support provide this already.

Closed Issues

Closed on - 2022-08-02 20:37:52 qpid-proton build fails when jsoncpp is present​

Closed on - 2022-08-02 20:37:52 RPATH/RUNPATH not set in qpid-proton libraries​

YODA Activity

Open Issues

​[FEATURE] doi versioning​

It is cumbersome not to be able to update my data files after they have been published.

​[FEATURE] Check ORCID for correctness in metadata form​

To prevent errors, it would be good to have a front end check against the standard orcid format (e.g. 0000-0001-2345-6789).

Closed Issues

Closed on - 2022-08-03 09:07:56 dynamic ‘group properties’ and ‘group members’ boxes placement​

I have a long list of categories in Yoda@WUR. When adding groups to a category all the way down, I have to scroll all the way up to add people and / or see group properties.

Gotta love the commentary on the closed ticket. A satisfied customer!

Closed on - 2022-08-03 13:50:59 [BUG] Tag in metadata removed after enter​

When typing in a tag and you hit enter once sometimes twice it will remove the tag entry

Only seems to happen if there is one tag available.

Closed on - 2022-08-04 06:29:51 [BUG] License metadata is not sent to DataCite​

Current Behavior
Access Level (https://guidelines.openaire.eu/en/latest/literature/field_accesslevel.html) is sent to DataCite in the rights field
Expected Behavior
License URI is sent to DataCite in the rights field

Closed on - 2022-07-28 09:02:54 Metadata of data objects also copied when data package is secured in vault [FEATURE]​

When a data package is submitted to the vault. The metadata of the collection is copied over, but the metadata of individual data objects within the data package are not copied along side with the data objects.

If you think someone else would appreciate this newsletter, they can sign up at https://theresource.metadata.school/​

No Yaks were shaved in the making of this newsletter. I was as surprised as you.

Don't miss what's next. Subscribe to The Resource:
Powered by Buttondown, the easiest way to start and grow your newsletter.