30Dec, 2023
    V12.1: Scanning inline

    …Now let’s tackle the last possible scanning method by Veeam in the new V12.1 version. It is the most advanced type that we are talking about, but with not too much information to share. Because all the magic happens in Veeam. As an admin you won’t notice a lot of it.

    The infra/architecture part:

    In short all the magic happens while taking a backup. The role of the proxy here is rather important. Because the component that is taking the backup will take an additional (cpu-)charge for these tasks. First best guesses are up to 15-25% more cpu power. Which is in most cases not a real issue as in most of the cases the proxies that are decently sized for a backup still have cpu-power left over… and in case of a virtual proxy this could mean that every 5 virtual proxys you cold (license-free) deploy another linux proxy to cope with the additional charge IF the backupwindow still needs to be the same.
    Consuming more CPU-power that is not available will result in more time required.

    The functional part:

    Enabling is again through the hamburger menu. There you’ll find the settings for the “malware detection”. Enabling the first checkbox will do the trick. The default value of the slider is “normal”, but this can be changed. “Extreme” will likely pop up some false positives overtime, and “low” will potentially miss some alerts in a general environment, but is better when encryption is being used…

    During the backup we use the proxy-component to tackle the data stream and read in the stream to scan the blocks. So in case we are using VMware, we ask VMware for the changed blocks (with the CBT), we will receive those changed blocks and will “sniff” into them and analyze it is a thread or not. This information is then being captured in a file (RIDX-file) on the proxy for every disk it is processing. When the job finished the file is being transferred to the VBRCatalog and we will kick in the Data-Analyzer service which will compare the current with the previous result.
    We will detect encrypted files (depending on the sensitivity level this will generate an event) and text artifacts (like the typical onion-links, and ransomware notes).

    Conclusion:

    With this final blog, we conclude looping over all the scanning possibilities in the new VBR 12.1. It is a LOT of information on this malware detection topic. And indeed, some of it might be completely irrelevant, because it is a general rule… IF we (Veeam or any other backupvendor) detects something, it is already too late… It means that your last line of defense has detected something that should have been detected earlier at your firewall, anti-virus, … or other security measures taken in the company…

    29Dec, 2023
    V12.1 Scanning almost inline…

    Why minimize such a great feature? Well, because it contains 2 parts… the brand new “inline” scanning feature works in 2 ways. Real inline (next blog) and what could be refferred as semi-inline…

    Let’s start with the requirements… you need some time and some space for a first full that is going to run. why? Simply because the major requirement is that guest indexing needs to be activated on the jobs.

    Enabling the guest file system indexing has also a small unknown disadvantage. It is actually a small file that is being generated at a full backup for the first time. Veeam will process that data of the machine and will index everything in that file. The main purpose is easy search-ability at Enterprise Manager… for instance for file level recoveries executed (likely with RBAC) by helpdesk engineers that needed to assist on specific requests.

    Enabling that for all servers is likely not going to happen in the field as you’ll need some additional rights on those servers and (if not mistaking) it can add up 100MB/1TB of processed data.

    So it will likely be active on certain “easy” accessible part of the production environment and not by default on the complete environment.
    What is then the purpose: the EXTRA pair of eyes at the sideline that will shout out when there is something wrong in production and your first line of defense didn’t shout out first…(or was on purpose de-activated)

    Now how can we detect those anomalities… Well at the moment the indexing-service has finished its work it will pass on a shout-out to the the data analyser service. This one will look over the index and will detect (compared with the previous scan) that “all of a sudden” stuff is (too rapidly) changing.

    I always try to compare everything with some sort of a fingerprint. The first time a full scan is done. the fingerprint that is virgin… imagine you work by default on a laptop, during the week our fingerprint will look like exactly the same every day :-)… however in the weekend you went out working in the garden en some scars appeared. This could trigger a “risk”… however it does not need to be… at the moment the risk is too high( you literally scraped half of your finger on a knife) then the alarm goes off.
    In IT this could be exactly the same, some changes are still ok, however a sudden bigger change might ring the horn! Example: part of the fileshare is encrypted = perfectly ok if it is classified data and/or the initial one has the same rate of normal vs encrypted data. However a sudden increase might mean 2 things: either some unwanted things happened, or (and never underestimate your clients) your end-users started to work 🙂 (or started a restructure/migration/…)

    The other advantage of the guest index file is that we can actually detect some new files were being added that are on the “blacklist”… That is also the reason why it is interesting to activate that update-mechanism ;). And not only new files are important, also the anomality about files suddenly disappearing (delete) or renamed are being detected.

    Now if you have V-One things are getting more obvious, plenty of alerts should have warned you already before we are running in the backupwindow.
    Regarding backupwindows that suddenly increased –> check the blog-article about suspicious file growths!

    So keep in mind…if we detect it (and not your virus-scan) it is already partially too late… and your backups are there to assist you on the restores!

    28Dec, 2023
    V12.1: Scanning (sidetrack): incident reported

    This blogpost is all about… what if…

    What I want to prevent with the anomalities and the inline scanning is that people start to believe that we (as a (very good and trustworthy) backupvendor) are going to give the impression that clients don’t need any security anymore… because… we can detect anomalities during the moment we take backups.
    Well.. that last part is true, but keep in mind… that means it is already too late! way too late!
    Your first line of defense is the one we don’t provide: firewalls, anti-virus, segmentation on networks,…

    Now… what if… your first line of defense actually detects something and were able to do some sort of a “ping” to us.
    Great news!!! We can accept those requests and it is brand new in V12.1. It is called “Veeam Incident API”. A very nice (and yet again simple) effective way to take yet another quick and last backup when a third-party notifies something wrong.

    The main concept was explained very well by my colleague in France. And I’ll be re-using his sample file.

    Adding this little check checkbox would allow us to take that additional last backup whenever some device detects something strange going on. So at that moment it might not be too late yet… but potentially the malware is only in its startup-phase and impact at that moment is just extremely low(and therefore still easy to extract with for instance secure restore).

    Now… how would that external device do that: it’s easy: talk to us/the API.

    I bet yet another colleague of me Steve (Stephan Herzig) made the test-powershell script. But those triggers can be executed by any other device. The most important is the timestamp and the indication of what it is all about (description and impacted machine). This generates an event and this event then captured by VBR to perform this quick backup.

    So let’s combine the forces of those gentlemens and trigger an event on my AD-server.

    So the main important part of the code is: 

    # Trigger a test event

#Building Body trigger a malware alert - Mimimun 2. Others are uuid or ipv6
$body_data = @{
    detectionTimeUtc = "$($utcNow)"
    machine = @{
                 fqdn          = "**********yourfqdn"
                 ipv4          = "**********yourIP"
                }
    details = "Steve's malware Detection Test script"
    engine  = "baptistetellier.fr made me do it"
}

Write-Host "Trigger malware detection event...." -ForegroundColor White
Write-Host ""
$appURI             = "/api/$APIversion/malwareDetection/events"
$body_json          = $body_data | ConvertTo-Json
Post-VeeamRestAPI-Audit -AppUri $appURI -Token $token -body $body_json

    They always say save the best for last… because in this example Steve’s script triggered the event (or in real live a security device)… but it actually pops up in the GUI as well. And when we dig a little deeper we’ll get the full details that were passed on to the API too:

    For those looking for that last screenshot(I was searching myself quite some time;)) … that is actually the “history-tab”. And with this little trick… I’ll leave the rest for the post of tomorrow… back to the real scanning – anomalities!

    27Dec, 2023
    V12.1: Scanning (part2): Post scanning

    To continue on the scanning part and to continue on how it was in the past compared to now let’s make a small comparison on the potential post-operational scanning.

    V12V12.1
    Surebackup scanSurebackup scan + verification and content scan
    Secure RestoreSecure restore
    Data integration APIData Integration API + Scan disk
    Index anomalities

    Let’s start with the added features. Surebackup job now allows to add a verification and content scan.

    It does NOT allow you to test the time to restore a machine, nor the ability to test the OS, but it allows you to scan the integrity of the filesystem and verify certain items are “not there”… example: some malware, some visa-card numbers…
    The good thing is that with this setup you can randomly select machines to get processed… pré 12.1 you ware obligated to create a fixed lab, and schedule wisely. Nowadays you could randomly have Veeam select a few jobs, process some in parallel, and thus easily do additional scanning on the complete environment on a monthly base.
    Like shown in the right screencap, I’ll be scanning my machines on a malware. Which is not very common, because if I were in that situation I had some unwanted visitors or behavior already in my production environment… and currently checking if I would be able to detect some more traces.
    I’m also performing the integrity check. Previously known as health-check. Both as post-processing option available in the job-settings, but also(very unknown) available on our ISO as standalone program.

    (after limiting the scope because of limited resources in my lab)
    The result is that the disks of the servers are being mounted by our mount server. Followed by actions on this mount (VeeamFLR-folder!).

    And if we are going to look after the running processes on our resource monitor in windows we can quickly discover some YARA -executables running. Keep track of the consumed CPU and additional diskload. Make sure that production is not being impacted.

    If all goes well, no harmful files were discovered. Now this was “just” 1 rule being used for scanning… it is possible however to combine rules with

    include "C:\Program Files\Veeam\Backup and Replication\Backup\YaraRules\file1.yara"
include "C:\Program Files\Veeam\Backup and Replication\Backup\YaraRules\file2.yara"

    That would allow you to make some kind of “mother-yara” file. Or yara-file dedicated for silver, gold, premium,… If you want to automate and include all the yara feel free to use explore the following post from my colleague Steve or use the dedicated script directly.
    Now we can proceed to the next potential scan for animalities during or partially after the backup was taken. That will be covered tomorrow…

    26Dec, 2023
    V12.1: Scanning (part1): how it was…

    In order to understand the scanning possibilities on V12.1 we must make a very clear distinction on “when” the scanning might happen. I’ll be focussing in the next few posts more on the post-scanning.

    pré-V12.1 (yes… I’ll teach you some french :-)) there were already post-scanning possibilities. But they were not very frequently used as they are not very known.
    One of the most known was the use of SureBackupJob with some added script on it. This allowed to boot up a server (or a portion of the environment), launch a scanning script in order to achieve something. More information and even an example can be discovered in our helpcenter.
    The others were less known/used:
    Secure restore that uses the same principle as the surebackup (mount everything temporarily, do something with it and push through in production). Perfect usecase:

    • Someone applied the right to be forgotten, a restore of an older version was required and prior pushing everything in production again we remove the person before pushing it in production.
    • You’ve been “attacked” or had an issue with a virus-outbreak. You use the isolated step in between to remove the malicious soft prior pushing it back in production (obviously makes more sense)

    Now the other one was the least known: data integration API. In short, what does it do? You can (ab-)use Veeam to publish a disk (like we always during in file level restore) and capture the data at C:\VeeamFLR\ on the mountserver. You could potentially mount it on ISCI and access it from another (powerful scanning) server. This will become a disk with read-only(content is coming from backups… and backupdata is not allowed to be “manipulated”). In my early days during a tech session someone asked why not… the answer is actually obvious :-)… otherwise I would be as a backupadmin thé perfect person to commit fraud and manipulate the logfiles in the backups of those financial servers… I would be rich and untraceable ;-).

    With V12.1 we cannot “just forget” everything, but we can use more function and even GUI. Just like the data integration API-method –> we just implemented this in the GUI as “publish disk“. It allows you to do more or less exactly the same in the GUI, and do the same operations.
    Scanning that: piece of cake… just mount and scan.

    Now why so much text and no images in this post? Because the basics are now covered and we can proceed on “guest file indexing” to jump to the “file system activity analysis” which is super-new in VBR. I’ll be covering the main advantages, but also the caveats… stay tuned for tomorrows post because: 1 post a day will keep the hacker away…

    20Dec, 2023
    Yet another copy

    What if… your CIO comes to your desk and requests a copy of some data for audit purposes…
    Source: some backupjob
    Target: somewhere on a Veeam-repository that is shared with an auditor
    OR(other usecase)… something is required to seed somewhere else in another geo over an untrustworthy line that would force you to manually restart (over and over) the copy.

    Well there are 2 ways to do this. An export data which is explained here: https://veeamclick.be/v12-1-an-unusual-export/

    or a simple “copy backup”… NOT a job that would do this permanently, but a one-time-hit-and-run.

    By selecting the “copy backup” option it would allow you to simply get another copy of the latest restore point into a new repository.

    Next to the export backup (that extracts a specific restore point) this is one of the most frequently forgotten features with many potential usecases.

    7Dec, 2023
    A blogpost a day…

    will keep the hacker away…

    To finish the year i’ll be posting every day a new blogpost.

    How will this work: easy, just keep a closer look at the blogs section as it will continue to grow every day.

    What are the main topics: security related and linked with 12.1; some insights on V-One, some new items on VB365 and much more.

    20Nov, 2023
    Veeam appliance (v2)

    For those who are following me a little… guess what… It looks like I inspired some people ;-).

    Next to Object First that has a very performant object storage “box” that can be used by Veeam when it leaves the factory there seems to be a new player in that market…

    Up to 3PB Veeam-ready appliance with built-in immutability. Excellent!

    Do you want to play around and discover how the Artesca works with Veeam? Then try those clickable demos for the creation of an account and immutable bucket.

    25Oct, 2023
    V12.1 an unusual export

    As the blogging-season officially started I wanted to take the “non-traditional” tour. A lot of my peers are making blogs on the main features like the inline detection or the scanning mechanism built in in our new version. And if we look at the clickdemo on our site it looks like I’ll be out of work soon :-).

    https://www.veeam.com/whats-new-backup-replication.html

    Sooooow…. let’s try an alternative approach and blog about the small nice things that are a little bit hidden under the surface. For instance my own preferred feature… It was added in VBR 9.5U4 for the first time and actually I requested this feature in my early days when I was still a client. The funny fact… I never used it in a real production as I’ve joined the Veeam-team when 9.5U4 came out :-). It’s called “export backup”.

    Where did it come from?

    For the “one time backup” or “just another to be sure backup” we have the feature called “quick backup“. A feature that is very frequently forgotten btw and also appears extremely likely on a vmce-exam!

    Now imagine you work in some sort of a project and when you start a clean instance it is called “version1”. During the development phase you go to “version2” after some days and you’ll need a new clean “version2” backup for whenever something happens with the environment (or during the development) that you’re not obligated to jump back all the way to V1… Quickbackup can do that for you… but what if… you’re on holiday or the dev-team has forgotten to tell you it was 2days ago…

    Export backup will pull out all information from the restore points and consolidate it in 1 new restore point. Since V12 you can assign a retention on it so that it gets deleted after x time.

    What is new?

    Not sooooo much, but for features like export backup, quickbackup, veeamzip, move and copy you’ll be able to assign your own dedicated policies. Assign names like “silver policy” instead of a fixed amount of choices with their fixed and assigned days/months/years.


    How to do it?

    Pretty simple, just change a json file. Either replace one of the possibilities, or add your own, but take note of the “,” !!! I’ve missed it and it took some time before I realised why it didn’t appear in the dropdown ;-).

    For beta-testers: yes, you did nothing wrong… no, it did not appear. Just reboot the server. It is planned to be fixed in GA 😉

    21Sep, 2023
    V12.1 or was it V12a

    And yes… the first babys-steps are taken. A release of the beta gave me more insights and I’m quite pleased with the work done by the teams.
    My attention was mostly attracted by the V-One integration in VBR (yesyes… exceptionally nice dashboard and metrics) that will give insights…

    So yes… in this tiny setup there is clearly still some work to do :-)… but I was also triggered by the “how would I get my score better” – question. And actually I was surprised by all the possible actions that were simply listed in the best-practice-analyser(that also suddenly changed icon).

    Next to the dash and the advice, some time ago a colleague asked me a question about the percentages as bottleneck (all good for another blogpost pretty soon). So I dared to ask that to our new AI-robot…

    Which actually gave me a pretty good answer. And this gives me hope, a lot of hope… currently I still need to answer quite a lot questions from clients that bumped into a small issue, could find where to tweak something or just had a generic question. I’m convinced that by promoting this chat I’ll be having less work :-). So my goal for 2024 is hereby defined. Train you to ask the right question, because the answer will be given and extremely likely very correct.

    Now if you think it is really all about monitoring and intelligence, then you’re all wrong. There is a lot, a lot more, but a lot more is going to be announced and get visual in the coming months. Stay tuned and discover about inline detection, scanning an publishing and so on!