August 2022 Edition of The Resource
Hello Reader, here is this month’s iRODS news and developments!
If you’re facing an issue with iRODS you’re not sure how to solve, please do drop me a line; if I’ve come across a solution or seen something relevant elsewhere, I’ll do my best to let you know. Or just drop me a mail to say ‘Hi’. Always nice to hear from people, particularly in these pandemic times!
I’d love your thoughts and feedback on how this newsletter could be better for you.
News
2022 UGM Videos are live!
Abstracts and slides at https://irods.org/ugm2022
TRiRODS Aug 2022 - Interns!
Refactor and modernize the main irodsServer - June Releford
Might also cover;
- Refactor and modernize the main irodsServer
- Add irodsServer configuration live reload
- AMQP Audit Rule Engine Plugin Investigation and iRODS C++ REST API Additions
- New Ticket Library
- iRODS Testing Environment Web Application
(I confess, I have not yet watched it)
RENCI looking for their next C++ Core Developer
Required: 4y degree, modern C++, a love of open source
Unofficial iRODS Docs are starting to take shape!
As mentioned in my lighting talk (not see it? Check out above!), a few people have started submitting documentation to the unofficial iRODS Documentation Project - you can too!
It being the holidays, I’ve not had as much time to contribute as I’d like, but it’s already got a few pages and three contributors!
Main Repository Activity
Open Issues
Remove pyparsing python module
Add query functions to new ticket administration library
This would give developers a way to query info about tickets.
Add existence check functions to new ticket administration library
Remote configuration reloading
The configuration reload feature doesn't cover any use cases with multiple instances.
Given that, it is desirable to add this to irods-grid so that it may be invoked across the zone, something like `irods-grid reload all`
Davrods and Dynamic Policy Enforcement Points
Davrods does not hit the actual put API, but performs an open/write/close. Please see this comment for details on which dynamic PEPs are engaged in the put operation: UtrechtUniversity/davrods#18 (comment)
The dynamic PEPs respond to actual API calls whereas the static PEPs are placed wherever the developer feels like it. In this case, the "oprType" can be set and rsDataObjClose understands this as having been a "put" and so the static PEP fires in a hard-coded location.
Adjust clang-format and clang-tidy configuration
Add option that instructs the server to track the time spent in rules
Providing such a feature would give administrators a way to see where time is spent regarding their policy and make informed decisions about how to improve it.
iRODS rule language KeyValuePair cannot have ‘size’ has key
Server side implementation of Metadata library not available in RodsAPIs
Sporadic test failures: prepareServerSharedMemory errors on server start sometimes
Add markdown file that documents how to compile iRODS from source
The iRODS repo does not provide any guidance on how to compile iRODS from source.
The project needs to introduce a new markdown file that explains how to compile iRODS. The markdown file should cover the following:
Dependencies
CMake options (e.g. CMAKE_INSTALL_PREFIX, CMAKE_BUILD_WITH_INSTALL_RPATH)
iRODS options (e.g. IRODS_UNIT_TESTS_BUILD)
Normal build steps (i.e. cmake . && ninja package)
Recommended way of compiling iRODS (i.e. irods_development_environment)
The top-level README file can point to this new markdown file.
Hurrah! This saves me from having to put something together for the Unofficial guide… :-)
add version number to icommands spOption (for use via ips)
The client string printed by ips does not currently include any version information.
Alas, spOption doesn’t appear to be documented (I swear it used to be?), so here’s the source code.
Remove .gitmodules
DataObjRename fails if similar named object exist in source collection
The definitely comes under the ‘well I learned something today’;
if you do a move and rename at the same time, for example,
'imv file1 coll1/file2', it will normally succeed if there's no conflicting
data-object name in the source collection (file2) but fail (giving error
CAT_NAME_EXISTS_AS_DATAOBJ) if there is, since, internally IRODS is doing
a rename and then a move. Please handle this by running multiple separate
'imv' commands.
Add more tests for METADATA_INCLUDED_KW(i.e. iput --metadata)
Merge filesystem.tpp with filesystem.hpp
Test downgrade of rodsadmin service account on a remote server
Adding a duplicate AVU should result in an error
This appears to be a behaviour change in 4.3;
Doing this will create a problem for iput --metadata. See #6409 for details.
This change in behavior is not good for existing deployments that rely on duplicate AVUs producing an error.
Replica in cache marked good after stage-to-cache returns an error
This relates to the S3 plugin and Glacier.
Tickets, Permissions, and Sharing data - A question about expectations
It's not uncommon to give another user access to data they don't own. There are two ways to do this:
Create a ticket for the target object and give the ticket to the user
Or, modify the permissions on the target object and then on the parent collection
The point of interest here is that using a ticket doesn't require modifying permissions on the parent collection. Great. The user can't look around the parent collection. All they can do is access the data associated with the ticket.
Q. Should modifying the permissions on the parent collection be necessary?
Q. Should modifying permissions on the target object result in the same behavior as using a ticket?
Add -f option to imv in order to allow overwrites
The rename API currently allows overwriting when FORCE_FLAG_KW is passed via the condInput of the dataObjInp_t.
imv does not expose a way to use this feature. We should add a -f option to allow for overwrites as the API already allows for.
icp cannot provide source resource or replica number
icp has the ability to specify a destination resource via the -R option. It does not have an option to specify a source resource or replica number for the source data object. For many iCommands, this is -S for source resource and -n for replica number.
As with other iCommands, -S and -n should be considered incompatible parameters.
The recursive flag -r should also be considered while implementing this feature: What happens if the specified source resource or replica number is not valid for every data object in a collection?
-R option in icp should be a directive, not a preference
Definitely in 4.3.0, likely earlier.
Observed behavior (including steps to reproduce, if applicable)
The resource specified by -R is ignored and the existing replica on ufs0 is overwritten:
Severe imkdir performance impact with inheritance enabled
Still under investigation. Adding an index for the columns ACL section may help, will report back.
Merge irods_logger.tpp with irods_logger.hpp
Add GitHub workflow for clang-tidy
Investigate how Coroutines can improve iRODS
test_parallel_transfer_engine on branch with apparently unrelated changes.
Replace Packstruct binary encoding (Pluggable Message Encoders/Decoders)
Investigation / Enhancement
The goal of this effort is to provide a binary encoding that is maintainable, efficient, extensible, versioned, documentable, and accessible to multiple programming languages.
We can achieve this by introducing pluggable binary encoders and decoders.
Potential Serialization Libraries
Flatbuffers
Cap'n Proto
Protocol Buffers
Cereal
MessagePack
Simple Binary Encoding (SBE)
Things to keep in mind
Backwards compatibility
Federation
Encoding negotiation
Investigate memory usage around rsGenQuery() and large database result sets
There is a possibility that this comes from the use of SpecficQueries (i.e. ones written by an admin and inserted with iadmin asq). The investigation continues!
Provide mapping between iRODS error codes and POSIX error codes
This is about categorizing iRODS error codes so that icommands and other clients return meaningful POSIX error codes instead of a random integer between 1 and 256.
This can be achieved using a map and a function. Bonus points for allowing other programming languages (e.g. Java, Python, Go, etc.) to consume the existing C/C++ mappings in a convenient manner.
Reference material:
https://man7.org/linux/man-pages/man3/errno.3.html
Investigate specific API endpoint for groupadmin operations
Investigate use of liburing
Delayed rules do not honor remote user ownership
The problem seems to be this: As of 579d0c2, delayed rule deletion is now correctly checking the permission level of the client user rather than the proxy user. The proxy user is always (or almost always) the service account rodsadmin account, which means that user will have permission to remove the delayed rule. Now that we are checking the client user, the usernames don't match because remote users always have the zone name appended but the delayed rule information is dropping the zone name from the username.
The solution, I think, is to write down the user name with the zone name in the delayed rule information in the catalog, and adjust the username check to account for this.
Rocky 9 released on July 14th
Unattended Install non-default irods home results in error
iRODS does not support changing the home collection during setup at this time. The setup script assumes certain things will be configured a particular way. This allows us to make assertions regarding correctness. After setup completes, you're free to change the home collection. To do that, you'd do the following:
Create a new collection using imkdir.
Open /var/lib/irods/.irods/irods_environment.json and set irods_home to the absolute (logical) path of the new collection.
With that said, allowing this in the future is a possibility.
Q. Is this something you need?
Q. Can you explain the motivation behind this?
itouch non existent file creates undeletable object
this is a known issue with itouch on the S3 resource plugin. We will look into this.
Add safe version of irods::at_scope_exit
iCommands do not disconnect on SIGINT
iCommands should handle SIGINT and disconnect from the server before aborting, if possible. This will prevent connections from staying open unnecessarily after the user has aborted the iCommand.
To be clear, connections are cleaned up by the server after the TCP connection times out, but this just handles the client-initiated terminations more gracefully.
Closed Issues
Closed on - 2022-08-16 13:54:55 Should that inheritance only affects files added to the directory after the inheritance bit is set?
Apparently this too is where I learned something - behaviour as documented.
Closed on - 2022-08-05 23:40:08 Remove problematic resources from mkresc
Misleading title, in my opinion - the reporter has somehow managed to make a resource with the same collection name as the homedir of the irods user. I wonder if that’s checked for in the iadmin mkresc - probably not, because someone might want to, because reasons…
Closed on - 2022-08-02 19:51:06 UseLibCXX: CMake version and LINK_LANGUAGE generator expression
Closed on - 2022-07-29 16:30:00 Server thread policy about threads
If you are interested in the interplay between the assorted configuration values that can determine the number of threads parallel transfer uses, this is a good issue to read through.
Closed on - 2022-08-02 19:41:33 Problem Linking CXX shared library libirods_common.so
Closed on - 2022-07-20 13:27:00 PAM auth does not work after upgrading to 4.3.0-1
Documentation is updated to address;
If you're using icommands, replace the use of "PAM" in your irods_environment.json file with "pam_password".
Gnnng. I’ll need to adjust all several hundred of our users configs when we upgrade. I hope the PR attached updated the docs for 4.3.0, but I suspect it will only be reflected in 4.3.1.
Closed on - 2022-07-05 15:13:51 Caught exception in migrate_delay_server
I think this means that the amqp plugin needs bumping to 4.3.0.1?
Further investigation has revealed that this only happens with 4.3.0.0 of the audit rule engine plugin installed and configured according to the training slide deck.
As this does not happen with a default configuration or with packages built from tip of main, I think this is actually an issue with the audit rule plugin and was fixed with irods/irods_rule_engine_plugin_audit_amqp#99.
Closed on - 2022-08-10 15:45:20 Set RUNPATH instead of RPATH
Python iRODS Client Activity
Open Issues
Nonexistent data object raises CollectionDoesNotExist
I think a decent summary of this issue is
PRC has historically done a query to see if the collection component of the requested new data object exists, and of course the Genquery is unable to distinguish between a collection it cannot write and one that doesn't exist.
Uploading data raises a KeyError -1205000 once the logical quota starts monitoring
4.2.10/11 reported. Seems to be an issue just with the Python client and logical quotas, as icommands reported running fine.
Large put() over federation leaves “valid” replicas of incorrect size and checksum when interrupted
This is actually illustrating an issue with < 4.2.9, rather than a Python client issue.
Large put() times-out with NetworkException: Could not receive server response
Large files (here tested with >= 50 GB) timing out when uploading to a replica resource based tree. Looks like some timeouts in the Python client somewhere rather than iRODS itself…
Retrieving large files causes unacceptable resource usage on catalog server
This is odd, and worrying if you are trying to use the Python iRODS Client for production work without having some kind of load balancing/throttling. Seems odd behaviour compared to the normal iput/iget but perhaps I am missing something?
During a large PRC put() both catalog and resource servers have normal (< 5% increase in both CPU and RAM) resource usage, but during a large PRC get() the catalog server's resource usage spikes strongly (nearly 100% CPU and RAM/swap) while the resource server's usage remains low. The spike is quicker with more threads and the transfer rate for get() is < 20% that of put() for the same data.
Add resource server redirect support to iput()/iget()
… and the prior issue makes more sense now; instead of passing off the data transfer to the Consumers, it’s funnelled through the Consumer that the client connected to instead - a behaviour that usually only happens with single threaded iget when using icommands.
NFSRODS Activity
Open Issues
access via nfsrods not working well.
The short version of this is that NFSRODS maps NFS UID’s to the local zone users, which doesn’t work well when there isn’t a corresponding one, but there is a Federated one of the same name.
Building nfsrods in an automation fails
Complete with PR and discussion of fixes.
icommands Activity
Open Issues
icp cannot copy files larger than 2 GB
Seems to be a regression in 4.3.0 and 4.2.11 icommands talking to a 4.2.8 Zone.
As 4.2.9 made a number of changes (it really should have been 4.3.0, but I can understand not wanting to push what is now 4.3.0 to be 4.4.0!), it sounds like Zones later than 4.2.8 need to be talked to with a similar client and vice versa.
Closed Issues
Closed on - 2022-08-05 11:10:39 iCommands 4.3.0 fails to run with PAM
The configuration value for enabling PAM in 4.3.0 has changed from "PAM" to "pam_password".
The docs are updated for 4.3.1.
Externals Activity
Open Issues
python3-pyodbc needed in EL7
EL7 (Centos 7, RHEL 7, and friends) needs a python3-pyodbc package. All other distros we support provide this already.
Closed Issues
Closed on - 2022-08-02 20:37:52 qpid-proton build fails when jsoncpp is present
Closed on - 2022-08-02 20:37:52 RPATH/RUNPATH not set in qpid-proton libraries
YODA Activity
Open Issues
[FEATURE] doi versioning
It is cumbersome not to be able to update my data files after they have been published.
[FEATURE] Check ORCID for correctness in metadata form
To prevent errors, it would be good to have a front end check against the standard orcid format (e.g. 0000-0001-2345-6789).
Closed Issues
Closed on - 2022-08-03 09:07:56 dynamic ‘group properties’ and ‘group members’ boxes placement
I have a long list of categories in Yoda@WUR. When adding groups to a category all the way down, I have to scroll all the way up to add people and / or see group properties.
Gotta love the commentary on the closed ticket. A satisfied customer!
Closed on - 2022-08-03 13:50:59 [BUG] Tag in metadata removed after enter
When typing in a tag and you hit enter once sometimes twice it will remove the tag entry
Only seems to happen if there is one tag available.
Closed on - 2022-08-04 06:29:51 [BUG] License metadata is not sent to DataCite
Current Behavior
Access Level (https://guidelines.openaire.eu/en/latest/literature/field_accesslevel.html) is sent to DataCite in the rights field
Expected Behavior
License URI is sent to DataCite in the rights field
Closed on - 2022-07-28 09:02:54 Metadata of data objects also copied when data package is secured in vault [FEATURE]
When a data package is submitted to the vault. The metadata of the collection is copied over, but the metadata of individual data objects within the data package are not copied along side with the data objects.
If you think someone else would appreciate this newsletter, they can sign up at https://theresource.metadata.school/
No Yaks were shaved in the making of this newsletter. I was as surprised as you.