Revolutions

Copilot for R

It was my great pleasure to present last week to the NYC Data Hackers on the topic of Copilot for R. If you haven't come across Copilot before, it's like an AI-based pair programmer that suggests new lines of code, and perhaps entire functions, based on context. In the presentation (video below) I showed how while I was editing in Visual Studio Code with Copilot enabled, it suggested tidyverse functions for cleaning a data set, and even the code for performing an analysis of variance:

I also took a behind-the-scenes look to see how Copilot uses Generative AI to make its suggestions. With the Azure OpenAI Service, you can access the underlying OpenAI Codex model directly, and generate code suggestions via its API. I showed an R script to access the OpenAI API directly using the httr2 package, and also provided an R function to call an OpenAI model. We also had some fun with the text-based GPT3 model to generate a poem about Python and to suggest a funny cat name token by token.

Thanks to everyone who attended the talk and asked such awesome questions! If you missed it, the video is above and all the code, links and resources can be found in the GitHub repo linked below.

GitHub (revodavid): Copilot for R

MRAN Time Machine will be retired on July 1

On July 1 2023, the MRAN website and the CRAN snapshot archive hosted therein will be retired, Microsoft announced last week. 

One purpose of MRAN was to distribute Microsoft R Open, which was discontinued in 2021. MRAN has also served since 2014 as the repository of a daily archive of CRAN packages. Some R users may still be using these static CRAN snapshots as a default CRAN repository for R scripts either directly, via the checkpoint package, or via older images of the rocker/r-ver container. If this affects you, you will need to find or create a new source of the package archive before MRAN's retirement.

The blog post linked below recommends using the miniCRAN package to download and save the package archives, a process documented in detail on Microsoft Learn. Commercial users of RStudio may also wish to explore the Posit Package Manager product from the creators of RStudio.

Microsoft Community Hub: Microsoft R Application Network retirement

 

Easy R Tutorials with Dev Containers

Ever written a tutorial for R — or simply shared some R code with someone — and been told that your code doesn't work on their machine? There's now an easy way to solve that problem: Dev Containers. Dev Containers are a standardized container format designed for interactive use. You can set up a Dev Container with the specific version of R, R packages, and other software to guarantee that your R code will run on their machine just as it did on yours. Dev Containers are 100% open source and integrated with Visual Studio Code, which provides a convenient environment for running your R scripts and notebooks. 

At the recent User!2022 conference, I gave a 20-minute presentation describing how Dev Containers work with R:

(Dev Containers also work with GitHub CodeSpaces, to create a cloud-based environment for running your R code in a single click. I demonstrated that at last week's RStudio::conf, and I'll post that video when it's available.)

If you'd like to set up a Dev Container for your own R project, the GitHub repository linked at the link below provides all the info you need.

GitHub (revodavid): Easy R Tutorials with Dev Containers

Using Microsoft365R with Shiny

by Hong Ooi

This article is a lightly-edited version of the "Microsoft365R and Shiny" vignette in the latest Microsoft365R release.

We describe how to incorporate Microsoft365R and interactive authentication with Azure Active Directory (AAD) into a Shiny web app. There are a few steps involved:

• Register your app with AAD
• Use the app ID to authenticate and get an OAuth token
• Pass the token to the Microsoft365R functions

App registration

The default Microsoft365R app registration only works when the package is used on a local machine; it does not support running in a remote server. Because of this, when you use Microsoft365R inside a Shiny app, you (or your friendly local sysadmin) must register that app in AAD.

The main things to set in your app registration are:

• The redirect URI of your app, ie, your user-facing site address. For example if your app is hosted in shinyapps.io, this would be a URL of the form https://youraccount.shinyapps.io/appname. If your app uses a special port number rather than the default port 443 for HTTPS, don’t forget to include that as well. It’s possible to set more than one redirect, so you can reuse a single app registration for multiple Shiny apps.

• The type of redirect, either native (mobile & desktop) or webapp. There are also other types of redirects, but these are the only ones relevant to R. The difference between a mobile & desktop and a webapp redirect is that you supply a client secret when authenticating with the latter, but not the former. It’s recommended to use a webapp redirect for a Shiny app, as the client secret helps prevent third parties from hijacking your app registration. The client secret is also set as part of the app registration.

• The intended audience of your app, ie, who is allowed to use it. This can be only members of your AAD tenant; members of any AAD tenant; or anyone with a Microsoft account (including personal accounts).

• The permissions required by your app. Refer to the app_registration.md file for the list of permissions Microsoft365R uses. You can omit any permissions that you don’t need if your app doesn’t use all of Microsoft365R’s functionality, eg if you don’t handle emails you can omit Mail.Send and Mail.ReadWrite.

The following pages at the AAD documentation will be helpful:

• A step-by-step guide to registering an app in the Azure portal.

• How to set permissions for an app. For a Shiny app, note that you want delegated permissions from the Microsoft Graph API, not application permissions.

Shiny code skeleton

Below is a basic app that logs the user in, retrieves their OneDrive, and lists the contents of the root folder.

One thing to note is that the regular Microsoft365R client functions like get_sharepoint_site, get_team etc are intended for use on a local machine. While they will still work when called in a web app, it’s a better idea to call the underlying R6 methods directly: Microsoft365R extends AzureGraph with several R6 classes and methods, which do the actual work of interacting with the Microsoft 365 REST API.

Here, we call the get_drive() method for the AzureGraph::az_user class, which retrieves the OneDrive for a user. For more information, see the online help page in R for the Microsoft365R “add_methods” topic: ?add_methods.

library(AzureAuth)
library(AzureGraph)
library(Microsoft365R)
library(shiny)

tenant <- "your-tenant-here"

# the application/client ID of your app registration
# - not to be confused with the 'object ID' or 'service principal ID'
app <- "your-app-id-here"

# the address/redirect URI of your app
# - AAD allows only HTTPS for non-localhost redirects, not HTTP
redirect <- "https://example.com/mysite"
port <- httr::parse_url(redirect)$port
options(shiny.port=if(is.null(port)) 443 else as.numeric(port))

# the client secret of your app registration
# - NEVER put secrets in code:
# - here we get it from an environment variable
# - unnecessary if you have a 'desktop & mobile' redirect
pwd <- Sys.getenv("EXAMPLE_SHINY_CLIENT_SECRET", "")
if(pwd == "") pwd <- NULL

# get the Graph permissions listed for the app, plus an ID token
resource <- c("https://graph.microsoft.com/.default", "openid")

# a simple UI: display the user's OneDrive
ui <- fluidPage(
    verbatimTextOutput("drv")
)

ui_func <- function(req)
{
    opts <- parseQueryString(req$QUERY_STRING)
    if(is.null(opts$code))
    {
        auth_uri <- build_authorization_uri(resource, tenant, app,
            redirect_uri=redirect, version=2)
        redir_js <- sprintf("location.replace(\"%s\");", auth_uri)
        tags$script(HTML(redir_js))
    }
    else ui
}

server <- function(input, output, session)
{
    opts <- parseQueryString(isolate(session$clientData$url_search))
    if(is.null(opts$code))
        return()

    token <- get_azure_token(resource, tenant, app, password=pwd,
       auth_type="authorization_code",
       authorize_args=list(redirect_uri=redirect), version=2,
       use_cache=FALSE, auth_code=opts$code)

    # display the contents of the user's OneDrive root folder
    drv <- ms_graph$
        new(token=token)$
        get_user()$
        get_drive()
    output$drv <- renderPrint(drv$list_files())
}

shinyApp(ui_func, server)

Microsoft365R: testers wanted

by Hong Ooi

Microsoft365R author here. I've just updated the package on GitHub with the following features:

• Add support for shared mailboxes to get_business_outlook(). To access a shared mailbox, supply one of the arguments shared_mbox_id, shared_mbox_name or shared_mbox_email specifying the ID, displayname or email address of the mailbox respectively.
• Add support for Teams chats (including one-on-one, group and meeting chats).
  • Use the list_chats() function to list the chats you're participating in, and the `get_chat()` function to retrieve a specific chat.
  • A chat object has class ms_chat, which has similar methods to a channel: you can send, list and retrieve messages, and list and retrieve members/attendees. One difference is that chats don't have an associated file folder, unlike channels.
• Updated the app permissions for Sharepoint to fix an access issue

However, at the moment I don't have access to a suitable tenant for testing these. This is a request for anyone whose tenant has approved the Microsoft365R app registration to install the development version and try using these features.

If you have previously used OneDrive, Teams and Outlook, everything should still work the same as before.

If you previously tried to use Sharepoint and got an authentication error, you should see a screen requesting you to grant Microsoft365R permission to access your Sharepoint site. You may or may not need an admin to approve the request, depending on your tenant's policies.

Please try it out and let me know of any problems! You can install the package from GitHub with

devtools::install_github("Azure/Microsoft365R")

AzureR update: new in May/June

by Hong Ooi

This is a summary of the updates to AzureR family of packages in May and June 2021.

AzureAuth

• Change the default caching behaviour to disable the cache if running inside Shiny.
• Update Shiny vignette to clean up redirect page after authenticating (thanks to Tyler Littlefield).
• Add a create_AzureR_dir function to create the caching directory manually. This can be useful not just for non-interactive sessions, but also Jupyter and R notebooks, which are not technically interactive in the sense that they cannot read user input from a console prompt.

AzureGraph

• Add enhanced support for the paging API. Many Graph calls that return lists of objects do so in pages: the first response contains a subset of the full list, along with a link to the next subset. AzureGraph now features a new ms_graph_pager R6 class, which is an iterator for the pages in the result. All list_* R6 methods have been rewritten to use this class, and have filter and n arguments to filter the result set and cap the number of results.
• Also add support for the batch request feature: you can pass multiple calls to the API with the graph_request R6 class and call_batch_endpoint function, and get the results back in a single response.
• See the Batching and paging vignette for more information on these APIs.
• Add list_users(), list_groups(), list_apps() and list_service_principals() methods to the main ms_graph client class.
• There is also a new Authentication vignette to guide users through the various ways to authenticate to Microsoft Graph.

AzureRMR

• Similar to AzureGraph, AzureRMR has a new Authentication vignette that details how to authenticate to Resource Manager. This replaces the old “Service principal” vignette.
• Update Resource Manager API version to “2021-04-01”; also update az_subscription$list_locations to handle the new response format in this API version.

AzureStor

• Add support for generating a service SAS. There is a new S3 generic get_service_sas with methods for az_storage and storage_endpoint objects, and a similar R6 method for az_storage objects. See ?sas for more information.
• Fix storage_save_rds and storage_load_rds to handle compression correctly. In particular, storage_load_rds should now correctly load files saved with saveRDS (#83).
• Fix a bug that caused list_blobs to fail when leases were present.
• Use a raw connection instead of a raw vector when calling readr::read_delim and read_csv2. This works around an issue introduced in readr 1.4.0 (#85, #86).
• Update client API version to “2020-04-08”. In particular, this allows specifying resource_type="d" when creating a service or user delegation SAS for blob storage.
• Add an optional service argument to storage_endpoint, to specify the service in question: blob, file, ADLS2, queue or table. This allows use of the generic endpoint function with URLs that don’t fit the usual pattern where the service is part of the hostname, eg custom domain names, IP addresses, etc.
  • For the same reason, remove the warning about an unrecognised endpoint URL from blob_endpoint, file_endpoint and adls_endpoint.

Microsoft365R

Microsoft365R has some significant new features in the latest version:

OneDrive/SharePoint

• Add a list_shared_items() method for the ms_drive class to access files and folders shared with you (#45).
• Allow getting drives for groups, sites and teams by name. The first argument to the get_drive() method for these classes is now drive_name; to get a drive by ID, specify the argument name explicitly: get_drive(drive_id=*)
• Add a by_item argument to the delete_item() method for drives and the delete() method for drive items (#21). This is to allow deletion of non-empty folders on SharePoint sites with data protection policies in place. Use with caution.

Outlook

• Add a search argument to the ms_outlook_folder$list_emails() method. The default is to search in the from, subject and body of the emails.

Teams

• Add list_members() and get_member() methods for teams and channels.
• Add support for @mentions in Teams channel messages (#26).

Other

• All list_* class methods now have filter and n arguments to filter the result set and cap the number of results, following the pattern in AzureGraph. The default values are filter=NULL and n=Inf. If n=NULL, an ms_graph_pager iterator object is returned instead to allow manual iteration over the results. Note that support for filtering in the underlying Graph API is somewhat uneven at the moment.
• Experimental read-only support for plans, contributed by Roman Zenka.
  • Add get_plan() and list_plans() methods to the az_group class. Note that only Microsoft 365 groups can have plans, not any other type of group.
  • To get the plan(s) for a site or team, call its get_group() method to retrieve the associated group, and then get the plan from the group.
  • A plan has methods to retrieve tasks and buckets, as well as plan details.

Microsoft365R 2.1.0 with Outlook support now on CRAN

by Hong Ooi

I’m happy to announce that Microsoft365R 2.1.0 is now on CRAN with Outlook email support! Here’s a quick summary of the new features:

• Send, reply to and forward emails, optionally composed with blastula or emayili
• Copy and move emails between folders
• Create, delete, copy and move folders
• Add, remove and download attachments

Here’s a sample of how to write an email using blastula:

library(Microsoft365R)

# 1st one is for your personal Microsoft account
# 2nd is for your work & school account
outl <- get_personal_outlook()
outlb <- get_business_outlook()

# compose an email with blastula
library(blastula)
bl_body <- "## Hello!

This is an email message that was generated by the blastula package.

We can use **Markdown** formatting with the `md()` function.

Cheers,

The blastula team"

bl_em <- compose_email(
    body=md(bl_body),
    footer=md("sent via Microsoft365R")
)
em <- outl$create_email(bl_em, subject="Hello from R",
                        to="[email protected]")

# add an attachment and send it
em$add_attachment("mydocument.docx")
em$send()

And on the other side, here’s a sample of how to work with the emails in your inbox:

# list the most recent emails in your inbox
emlst <- outl$list_emails()

# get the most recent email
em <- emlst[[1]]

# list and download attachments
em$list_attachments()
em$download_attachment("mydatafile.csv")

# reply to it
em$create_reply("Replying from R")$send()

# list the folders in your account
outl$list_folders()

# move the move recent email to a given folder
folder <- outl$get_folder("My project folder")
em$move(folder)

In addition, this release fixes a bug in the list_files() method for OneDrive/Sharepoint drives and drive items, and adds the ability to create nested drive folders in one call.

Please note that if you’re using one of the workarounds mentioned in the authentication vignette, they won’t work with Outlook. You’ll need to get the Microsoft365R app approved for your tenant, or alternatively, if you have admin rights you can create your own tenant with the required permissions. (This applies if you’re using Microsoft365R at work; if you’re using it at home on your personal account, you shouldn’t have any problems.)

If you have any feedback or comments, you can email me or open an issue at the repo.

Outlook client support in Microsoft365R available for beta test

by Hong Ooi

This is an announcement that a beta Outlook email client is now part of the Microsoft365R package. You can install it from the GitHub repository with:

devtools::install_github("Azure/Microsoft365R")

The client provides the following features:

• Send, reply to and forward emails, optionally composed with blastula or emayili
• Copy and move emails between folders
• Create, delete, copy and move folders
• Add, remove and download attachments

The plan is to submit this to CRAN sometime next month, after a period of public testing. Please give it a try and give me your feedback: either via email or by opening an issue at the repo.

Here’s a small sample of the client in action (taken from the README):

library(Microsoft365R)

# 1st one is for your personal Microsoft account
# 2nd is for your work & school account
outl <- get_personal_outlook()
outlb <- get_business_outlook()

# compose an email with blastula
library(blastula)
bl_body <- "## Hello!

This is an email message that was generated by the blastula package.

We can use **Markdown** formatting with the `md()` function.

Cheers,

The blastula team"

bl_em <- compose_email(
    body=md(bl_body),
    footer=md("sent via Microsoft365R")
)
em <- outl$create_email(bl_em, subject="Hello from R",
                        to="[email protected]")

# add an attachment and send it
em$add_attachment("mydocument.docx")
em$send()

# list the most recent emails in your inbox
emlst <- outl$list_emails()

# reply to the most recent email
emlst[[1]]$
    create_reply("Replying from R")$
    send()

Please note though, that if you’re using one of the workarounds mentioned in the authentication vignette, they won’t work with Outlook. You’ll need to get the Microsoft365R app approved for your tenant, or alternatively, if you have admin rights you can create your own tenant with the required permissions. (This applies if you’re using Microsoft365R at work; if you’re using it at home on your personal account, you shouldn’t have any problems.)

Teams support in Microsoft365R

by Hong Ooi

I’m happy to announce that version 2.0 of Microsoft365R, the R interface to Microsoft 365, is now on CRAN! This version adds support for Microsoft Teams, a much-requested feature.

To access a team in Microsoft Teams, use the get_team() function and provide the team name or ID. You can also list the teams you’re in with list_teams(). These return objects of R6 class ms_team, which has methods for working with channels and drives.

list_teams()
team <- get_team("My team")

# list the channels in a team (including your private channels)
team$list_channels()

# get the primary channel for a team
team$get_channel()

# get a specific channel
team$get_channel("My channel")

# drives for a team
team$list_drives()
team$get_drive()

A drive is an ms_drive object, so if you’re already using Microsoft365R to interface with OneDrive and SharePoint document libraries, you already know how to use a team’s drives. Each team will generally have at least one drive, and possibly two: the default “Shared Documents” drive, which is where uploaded files are stored, and the “Teams Wiki Data” drive, if the team has a wiki. Each team channel will usually also have an associated folder in each drive.

drv <- team$get_drive()

# one folder per channel
drv$list_files()

# upload will appear in Files tab of "My channel" in the Teams client
drv$upload_file("myfile.csv", "My channel/myfile.csv")

Channels

A team object has methods for listing, retrieving, creating and deleting channels. However you should not create and delete channels unnecessarily, since Teams tracks all channels ever created, even after you delete them. In turn, a channel object has methods for listing and sending messages, and uploading and deleting files.

Channel messages

Teams channels are semi-threaded. Getting the list of messages for a channel retrieves only the first message in each thread; to get an entire thread, you get the starting message and then retrieve the replies to it. Note that channels don’t have nested replies, so you can’t reply to a reply—only to the starting message.

The body of a message is part of the list of properties returned from the host, and can be found in the properties field of the object. Other properties include metadata such as the author, date, list of attachments, etc.

chan <- team$get_channel()

# retrieve most recent messages from the server
msgs <- chan$list_messages()

# get the latest message by ID
msg <- chan$get_message(msgs[[1]]$properties$id)

# body of the message
msg$properties$body

# 10 most recent replies
repl_list <- msg$list_replies(n=10)

# body of an individual reply
repl_list[[1]]$properties$body

You can send a message to a channel as plain text (the default) or HTML. A message can also include attachments and inline images.

# sending messages to a channel
chan$send_message("Hello from R")
chan$send_message(
    "<div>Hello from <em>R</em></div>", content_type="html")

# attachments and inline images
chan$send_message("Hello with attachments",
    attachments=c("intro.md", "myfile.csv"))
chan$send_message("",
    content_type="html", inline="graph.png")

# send a reply to a message
msg <- chan$send_message("Starting a new thread in R")
msg$send_reply("Reply from R")

Currently, Microsoft365R only supports messaging in channels. Support for chats between individuals may come later.

Channel files

Uploading a file to a channel will place it in the channel’s drive folder. The channel object itself provides convenience functions to list, upload and download files. It also provides a get_folder() method to retrieve the folder for the channel, as an ms_drive_item object; this object has more general methods for working with files.

# files for the channel
chan$list_files()

# upload a file to the channel
chan$upload_file("myfile.docx")

# open the uploaded document for editing in Word Online
chan_folder <- chan$get_folder()
item <- chan_folder$get_item("myfile.docx")
item$open()

# download it again
item$download(overwrite=TRUE)

Providing Feedback

Let us know how this works for you! You can provide feedback and make feature requests by opening an issue at the repo, or by emailing me at hongooi73 (@) gmail.com.

Microsoft365R: an R interface to the Microsoft 365 suite

I’m very happy to announce Microsoft365R, a package for working with the Microsoft 365 (formerly known as Office 365) suite of cloud services. Microsoft365R extends the interface to the Microsoft Graph API provided by the AzureGraph package to provide a lightweight yet powerful interface to SharePoint and OneDrive, with support for Teams and Outlook soon to come.

Microsoft365R is now available on CRAN, or you can install the development version from GitHub with devtools::install_github("Azure/Microsoft365R").

Authentication

The first time you call one of the Microsoft365R functions (see below), it will use your Internet browser to authenticate with Azure Active Directory (AAD), in a similar manner to other web apps. You will get a dialog box asking for permission to access your information.

Microsoft365R is registered as an app in the “aicatr” AAD tenant. Because it needs read/write access to groups and SharePoint sites, you’ll need an admin to grant it access to your tenant. Alternatively, if the environment variable CLIMICROSOFT365_AADAPPID is set, Microsoft365R will use its value as the app ID for authenticating; or you can specify the app ID as an argument when calling the functions below. See also this issue at the GitHub repo for some possible workarounds.

OneDrive

To access your personal OneDrive, call the personal_onedrive() function, and to access OneDrive for Business call business_onedrive(). Both functions return an R6 client object of class ms_drive, which has methods for working with files and folders. Note that OneDrive for Business is technically part of SharePoint, and requires a Microsoft 365 Business subscription.

od <- personal_onedrive()
odb <- business_onedrive(tenant="mycompany")

# use the device code authentication flow in RStudio Server
od <- personal_onedrive(auth_type="device_code")

# list files and folders
od$list_items()
od$list_items("Documents")

# upload and download files
od$download_file("Documents/myfile.docx")
od$upload_file("somedata.xlsx")

# create a folder
od$create_folder("Documents/newfolder")

You can open a file or folder in your browser with the open_item() method. For example, a Word document or Excel spreadsheet will open in Word or Excel Online, and a folder will be shown in OneDrive.

od$open_item("Documents/myfile.docx")

You can get and set the metadata properties for a file or folder with get_item_properties() and set_item_properties(). For the latter, provide the new properties as named arguments to the method. Not all properties can be changed; some, like the file size and last modified date, are read-only. You can also retrieve an object representing the file or folder with get_item(), which has methods appropriate for drive items.

od$get_item_properties("Documents/myfile.docx")

# rename a file -- version control via filename is bad, mmkay
od$set_item_properties("Documents/myfile.docx", name="myfile version 2.docx")

# alternatively, you can call the file object's update() method
item <- od$get_item("Documents/myfile.docx")
item$update(name="myfile version 2.docx")

SharePoint

To access a SharePoint site, use the sharepoint_site() function and provide the site URL or ID.

site <- sharepoint_site("https://myaadtenant.sharepoint.com/sites/my-site-name")

The client object has methods to retrieve drives (document libraries) and lists. To show all drives in a site, use the list_drives() method, and to retrieve a specific drive, use get_drive(). Each drive is an object of class ms_drive, just like the OneDrive clients above.

# list of all document libraries under this site
site$list_drives()

# default document library
drv <- site$get_drive()

# same methods as for OneDrive
drv$list_items()
drv$open_item("teamproject/plan.xlsx")

To show all lists in a site, use the get_lists() method, and to retrieve a specific list, use get_list() and supply either the list name or ID.

site$get_lists()
lst <- site$get_list("my-list")

You can retrieve the items in a list as a data frame, with list_items(). This has arguments filter and select to do row and column subsetting respectively. filter should be an OData expression provided as a string, and select should be a string containing a comma-separated list of columns. Any column names in the filter expression must be prefixed with fields/ to distinguish them from item metadata.

# return a data frame containing all list items
lst$list_items()

# get subset of rows and columns
lst$list_items(
    filter="startsWith(fields/firstname, 'John')",
    select="firstname,lastname,title"
)

Finally, you can retrieve subsites with list_subsites() and get_subsite(). These also return SharePoint site objects, so all the methods above are available for a subsite.

Future plans

Currently, Microsoft365R supports OneDrive and SharePoint Online; future updates will add the ability to post to Teams channels and send emails via Outlook. You can also provide feedback and make feature requests by opening an issue at the repo, or by emailing me at hongooi73 (@) gmail.com.